Introducing the Cloudflare Radar Internet Quality Page

2023-06-23 David Belson

Post Syndicated from David Belson original http://blog.cloudflare.com/introducing-radar-internet-quality-page/

Internet connections are most often marketed and sold on the basis of "speed", with providers touting the number of megabits or gigabits per second that their various service tiers are supposed to provide. This marketing has largely been successful, as most subscribers believe that "more is better”. Furthermore, many national broadband plans in countries around the world include specific target connection speeds. However, even with a high speed connection, gamers may encounter sluggish performance, while video conference participants may experience frozen video or audio dropouts. Speeds alone don't tell the whole story when it comes to Internet connection quality.

Additional factors like latency, jitter, and packet loss can significantly impact end user experience, potentially leading to situations where higher speed connections actually deliver a worse user experience than lower speed connections. Connection performance and quality can also vary based on usage – measured average speed will differ from peak available capacity, and latency varies under loaded and idle conditions.

The new Cloudflare Radar Internet Quality page

A little more than three years ago, as residential Internet connections were strained because of the shift towards working and learning from home due to the COVID-19 pandemic, Cloudflare announced the speed.cloudflare.com speed test tool, which enabled users to test the performance and quality of their Internet connection. Within the tool, users can download the results of their individual test as a CSV, or share the results on social media. However, there was no aggregated insight into Cloudflare speed test results at a network or country level to provide a perspective on connectivity characteristics across a larger population.

Today, we are launching these long-missing aggregated connection performance and quality insights on Cloudflare Radar. The new Internet Quality page provides both country and network (autonomous system) level insight into Internet connection performance (bandwidth) and quality (latency, jitter) over time. (Your Internet service provider is likely an autonomous system with its own autonomous system number (ASN), and many large companies, online platforms, and educational institutions also have their own autonomous systems and associated ASNs.) The insights we are providing are presented across two sections: the Internet Quality Index (IQI), which estimates average Internet quality based on aggregated measurements against a set of Cloudflare & third-party targets, and Connection Quality, which presents peak/best case connection characteristics based on speed.cloudflare.com test results aggregated over the previous 90 days. (Details on our approach to the analysis of this data are presented below.)

Users may note that individual speed test results, as well as the aggregate speed test results presented on the Internet Quality page will likely differ from those presented by other speed test tools. This can be due to a number of factors including differences in test endpoint locations (considering both geographic and network distance), test content selection, the impact of “rate boosting” by some ISPs, and testing over a single connection vs. multiple parallel connections. Infrequent testing (on any speed test tool) by users seeking to confirm perceived poor performance or validate purchased speeds will also contribute to the differences seen in the results published by the various speed test platforms.

And as we announced in April, Cloudflare has partnered with Measurement Lab (M-Lab) to create a publicly-available, queryable repository for speed test results. M-Lab is a non-profit third-party organization dedicated to providing a representative picture of Internet quality around the world. M-Lab produces and hosts the Network Diagnostic Tool, which is a very popular network quality test that records millions of samples a day. Given their mission to provide a publicly viewable, representative picture of Internet quality, we chose to partner with them to provide an accurate view of your Internet experience and the experience of others around the world using openly available data.

Connection speed & quality data is important

While most advertisements for fixed broadband and mobile connectivity tend to focus on download speeds (and peak speeds at that), there’s more to an Internet connection, and the user’s experience with that Internet connection, than that single metric. In addition to download speeds, users should also understand the upload speeds that their connection is capable of, as well as the quality of the connection, as expressed through metrics known as latency and jitter. Getting insight into all of these metrics provides a more well-rounded view of a given Internet connection, or in aggregate, the state of Internet connectivity across a geography or network.

The concept of download speeds are fairly well understood as a measure of performance. However, it is important to note that the average download speeds experienced by a user during common Web browsing activities, which often involves the parallel retrieval of multiple smaller files from multiple hosts, can differ significantly from peak download speeds, where the user is downloading a single large file (such as a video or software update), which allows the connection to reach maximum performance. The bandwidth (speed) available for upload is sometimes mentioned in ISP advertisements, but doesn’t receive much attention. (And depending on the type of Internet connection, there’s often a significant difference between the available upload and download speeds.) However, the importance of upload came to the forefront in 2020 as video conferencing tools saw a surge in usage as both work meetings and school classes shifted to the Internet during the COVID-19 pandemic. To share your audio and video with other participants, you need sufficient upload bandwidth, and this issue was often compounded by multiple people sharing a single residential Internet connection.

Latency is the time it takes data to move through the Internet, and is measured in the number of milliseconds that it takes a packet of data to go from a client (such as your computer or mobile device) to a server, and then back to the client. In contrast to speed metrics, lower latency is preferable. This is especially true for use cases like online gaming where latency can make a difference between a character’s life and death in the game, as well as video conferencing, where higher latency can cause choppy audio and video experiences, but it also impacts web page performance. The latency metric can be further broken down into loaded and idle latency. The former measures latency on a loaded connection, where bandwidth is actively being consumed, while the latter measures latency on an “idle” connection, when there is no other network traffic present. (These specific loaded and idle definitions are from the device’s perspective, and more specifically, from the speed test application’s perspective. Unless the speed test is being performed directly from a router, the device/application doesn't have insight into traffic on the rest of the network.) Jitter is the average variation found in consecutive latency measurements, and can be measured on both idle and loaded connections. A lower number means that the latency measurements are more consistent. As with latency, Internet connections should have minimal jitter, which helps provide more consistent performance.

Our approach to data analysis

The Internet Quality Index (IQI) and Connection Quality sections get their data from two different sources, providing two different (albeit related) perspectives. Under the hood they share some common principles, though.

IQI builds upon the mechanism we already use to regularly benchmark ourselves against other industry players. It is based on end user measurements against a set of Cloudflare and third-party targets, meant to represent a pattern that has become very common in the modern Internet, where most content is served from distribution networks with points of presence spread throughout the world. For this reason, and by design, IQI will show worse results for regions and Internet providers that rely on international (rather than peering) links for most content.

IQI is also designed to reflect the traffic load most commonly associated with web browsing, rather than more intensive use. This, and the chosen set of measurement targets, effectively biases the numbers towards what end users experience in practice (where latency plays an important role in how fast things can go).

For each metric covered by IQI, and for each ASN, we calculate the 25th percentile, median, and 75th percentile at 15 minute intervals. At the country level and above, the three calculated numbers for each ASN visible from that region are independently aggregated. This aggregation takes the estimated user population of each ASN into account, biasing the numbers away from networks that source a lot of automated traffic but have few end users.

The Connection Quality section gets its data from the Cloudflare Speed Test tool, which exercises a user's connection in order to see how well it is able to perform. It measures against the closest Cloudflare location, providing a good balance of realistic results and network proximity to the end user. We have a presence in 285 cities around the world, allowing us to be pretty close to most users.

Similar to the IQI, we calculate the 25th percentile, median, and 75th percentile for each ASN. But here these three numbers are immediately combined using an operation called the trimean — a single number meant to balance the best connection quality that most users have, with the best quality available from that ASN (users may not subscribe to the best available plan for a number of reasons).

Because users may choose to run a speed test for different motives at different times, and also because we take privacy very seriously and don’t record any personally identifiable information along with test results, we aggregate at 90-day intervals to capture as much variability as we can.

At the country level and above, the calculated trimean for each ASN in that region is aggregated. This, again, takes the estimated user population of each ASN into account, biasing the numbers away from networks that have few end users but which may still have technicians using the Cloudflare Speed Test to assess the performance of their network.

Navigating the Internet Quality page

The new Internet Quality page includes three views: Global, country-level, and autonomous system (AS). In line with the other pages on Cloudflare Radar, the country-level and AS pages show the same data sets, differing only in their level of aggregation. Below, we highlight the various components of the Internet Quality page.

Global

The top section of the global (worldwide) view includes time series graphs of the Internet Quality Index metrics aggregated at a continent level. The time frame shown in the graphs is governed by the selection made in the time frame drop down at the upper right of the page, and at launch, data for only the last three months is available. For users interested in examining a specific continent, clicking on the other continent names in the legend removes them from the graph. Although continent-level aggregation is still rather coarse, it still provides some insight into regional Internet quality around the world.

Further down the page, the Connection Quality section presents a choropleth map, with countries shaded according to the values of the speed, latency, or jitter metric selected from the drop-down menu. Hovering over a country displays a label with the country’s name and metric value, and clicking on the country takes you to the country’s Internet Quality page. Note that in contrast to the IQI section, the Connection Quality section always displays data aggregated over the previous 90 days.

Country-level

Within the country-level page (using Canada as an example in the figures below), the country’s IQI metrics over the selected time frame are displayed. These time series graphs show the median bandwidth, latency, and DNS response time within a shaded band bounded at the 25th and 75th percentile and represent the average expected user experience across the country, as discussed in the Our approach to data analysis section above.

Below that is the Connection Quality section, which provides a summary view of the country’s measured upload and download speeds, as well as latency and jitter, over the previous 90 days. The colored wedges in the Performance Summary graph are intended to illustrate aggregate connection quality at a glance, with an “ideal” connection having larger upload and download wedges and smaller latency and jitter wedges. Hovering over the wedges displays the metric’s value, which is also shown in the table to the right of the graph.

Below that, the Bandwidth and Latency/Jitter histograms illustrate the bucketed distribution of upload and download speeds, and latency and jitter measurements. In some cases, the speed histograms may show a noticeable bar at 1 Gbps, or 1000 ms (1 second) on the latency/jitter histograms. The presence of such a bar indicates that there is a set of measurements with values greater than the 1 Gbps/1000 ms maximum histogram values.

Autonomous system level

Within the upper-right section of the country-level page, a list of the top five autonomous systems within the country is shown. Clicking on an ASN takes you to the Performance page for that autonomous system. For others not displayed in the top five list, you can use the search bar at the top of the page to search by autonomous system name or number. The graphs shown within the AS level view are identical to those shown at a country level, but obviously at a different level of aggregation. You can find the ASN that you are connected to from the My Connection page on Cloudflare Radar.

Exploring connection performance & quality data

Digging into the IQI and Connection Quality visualizations can surface some interesting observations, including characterizing Internet connections, and the impact of Internet disruptions, including shutdowns and network issues. We explore some examples below.

Characterizing Internet connections

Verizon FiOS is a residential fiber-based Internet service available to customers in the United States. Fiber-based Internet services (as opposed to cable-based, DSL, dial-up, or satellite) will generally offer symmetric upload and download speeds, and the FiOS plans page shows this to be the case, offering 300 Mbps (upload & download), 500 Mbps (upload & download), and “1 Gig” (Verizon claims average wired speeds between 750-940 Mbps download / 750-880 Mbps upload) plans. Verizon carries FiOS traffic on AS701 (labeled UUNET due to a historical acquisition), and in looking at the bandwidth histogram for AS701, several things stand out. The first is a rough symmetry in upload and download speeds. (A cable-based Internet service provider, in contrast, would generally show a wide spread of download speeds, but have upload speeds clustered at the lower end of the range.) Another is the peaks around 300 Mbps and 750 Mbps, suggesting that the 300 Mbps and “1 Gig” plans may be more popular than the 500 Mbps plan. It is also clear that there are a significant number of test results with speeds below 300 Mbps. This is due to several factors: one is that Verizon also carries lower speed non-FiOS traffic on AS701, while another is that erratic nature of in-home WiFi often means that the speeds achieved on a test will be lower than the purchased service level.

Traffic shifts drive latency shifts

On May 9, 2023, the government of Pakistan ordered the shutdown of mobile network services in the wake of protests following the arrest of former Prime Minister Imran Khan. Our blog post covering this shutdown looked at the impact from a traffic perspective. Within the post, we noted that autonomous systems associated with fixed broadband networks saw significant increases in traffic when the mobile networks were shut down – that is, some users shifted to using fixed networks (home broadband) when mobile networks were unavailable.

Examining IQI data after the blog post was published, we found that the impact of this traffic shift was also visible in our latency data. As can be seen in the shaded area of the graph below, the shutdown of the mobile networks resulted in the median latency dropping about 25% as usage shifted from higher latency mobile networks to lower latency fixed broadband networks. An increase in latency is visible in the graph when mobile connectivity was restored on May 12.

Bandwidth shifts as a potential early warning sign

On April 4, UK mobile operator Virgin Media suffered several brief outages. In examining the IQI bandwidth graph for AS5089, the ASN used by Virgin Media (formerly branded as NTL), indications of a potential problem are visible several days before the outages occurred, as median bandwidth dropped by about a third, from around 35 Mbps to around 23 Mbps. The outages are visible in the circled area in the graph below. Published reports indicate that the problems lasted into April 5, in line with the lower median bandwidth measured through mid-day.

Submarine cable issues cause slower browsing

On June 5, Philippine Internet provider PLDT Tweeted an advisory that noted “One of our submarine cable partners confirms a loss in some of its internet bandwidth capacity, and thus causing slower Internet browsing.” IQI latency and bandwidth graphs for AS9299, a primary ASN used by PLDT, shows clear shifts starting around 06:45 UTC (14:45 local time). Median bandwidth dropped by half, from 17 Mbps to 8 Mbps, while median latency increased by 75% from 37 ms to around 65 ms. 75th percentile latency also saw a significant increase, nearly tripling from 63 ms to 180 ms coincident with the reported submarine cable issue.

Conclusion

Making network performance and quality insights available on Cloudflare Radar supports Cloudflare’s mission to help build a better Internet. However, we’re not done yet – we have more enhancements planned. These include making data available at a more granular geographical level (such as state and possibly city), incorporating AIM scores to help assess Internet quality for specific types of use cases, and embedding the Cloudflare speed test directly on Radar using the open source JavaScript module.

In the meantime, we invite you to use speed.cloudflare.com to test the performance and quality of your Internet connection, share any country or AS-level insights you discover on social media (tag @CloudflareRadar on Twitter or @[email protected] on Mastodon), and explore the underlying data through the M-Lab repository or the Radar API.

The SS Eastland Disaster

2023-06-23 The History Guy: History Deserves to Be Remembered

Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=4a5mEEFVcQY

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

2023-06-23 Bradley M. Kuhn

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2023/06/23/rhel-red-hat-gpl.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This article was originally published primarily as a response
to IBM’s
Red Hat’s change to no longer publish complete, corresponding source
(CCS) for RHEL and the
prior discontinuation of CentOS Linux (which are related events, as
described below). We hope that this will serve as a comprehensive
document that discusses the history of Red Hat’s RHEL business model,
the related source code provisioning, and the GPL compliance issues with RHEL.

For approximately twenty years, Red Hat (now a fully owned subsidiary of
IBM) has experimented with building a business model for operating system deployment and
distribution that looks, feels, and acts like a proprietary one, but
nonetheless complies with the GPL and other standard copyleft
terms. Software rights activists,
including SFC, have spent decades talking to Red Hat and its
attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts
disaster and is actively unfriendly to
community-oriented Free and Open Source Software (FOSS). These pleadings,
discussions, and encouragements have, as far as we can tell, been heard and
seriously listened to by key members of Red Hat’s legal and OSPO
departments, and even by key C-level executives, but they have ultimately been rejected
and ignored — sometimes even with a “fine, then sue us for GPL
violations” attitude. Activists have found this discussion
frustrating, but kept the nature and tenure of these discussions as an
“open secret” until now because we all had hoped that Red Hat’s behavior
would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even
worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL’s business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM’s Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how
many copies of RHEL are actually installed (see §10) — presumably for the
purpose of Red Hat getting the information they need to decide
whether to “fire” the customer.

Red Hat’s lawyers clearly take the position that this business model complies with the GPL (though we aren’t so sure), on grounds that that nothing in the GPL agreements requires an entity
keep a business relationship with any other entity. They have further argued that such business
relationships can be terminated based on any behaviors — including
exercising rights guaranteed by the GPL agreements. Whether that
analysis is correct is a matter of intense debate, and likely only a court
case that disputed this particular issue would yield a definitive answer
on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today,
in copyleft expert circles, whether this
model itself violates GPL. There is, however, no doubt that this
provision is not in the spirit of the GPL agreements. The RHEL business
model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL
business model remains, to our knowledge, rather unique in the software
industry. IBM’s Red Hat definitely deserves credit for so carefully
constructing their business model such that it has spent most of the last
two decades in murky territory of “probably not violating the
GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the
line of GPL compliance is that violations can and do happen — since
even a minor deviation from the business model clearly violates the GPL
agreements. Pre-IBM Red Hat deserves a certain amount of credit, as
SFC is aware of only two documented incidents of GPL violations that have
occurred since 2006 regarding the RHEL business model. We’ve decided to
share some general details of these violations for the purpose of
explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we’ll
call Company A), who both used RHEL internally and also built
public-facing Linux-based products, decided to create a consumer-facing
product (which we’ll call Product P) based primarily on CentOS Linux,
but P included a few packages built from RHEL sources. Company A
did not seek nor ask for support or update services for this separate
Product P. Red Hat later became aware that Product P contained
some part of RHEL, and Red Hat demanded royalty payments for Product
P. Red Hat threatened to revoke the support and update
services on Company A‘s internal RHEL servers if such royalties were
not paid.

Since Company A was powerful and had good lawyers and savvy
business development staff, they did not acquiesce. Company A ultimately
continued (to our knowledge) on as a RHEL customer for their internal
servers and continued selling Product P without royalty payments. Nevertheless, a
demand for royalties for distribution is clearly a violation as that demand creates a
“further restriction” on the permissions granted by GPL. As
stated in GPLv3:

You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore
violated the GPL. The violation was resolved since no royalty was paid
and Company A faced no consequences. SFC learned of
the incident later, and informed Red Hat that the past royalty demand was
a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree
such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific
non-USA country, was requiring that any customer who lowered the number of
RHEL machines under service contract with Red Hat sign an
additional agreement. This additional agreement promised that the customer
had deleted every copy of RHEL in their entire organization other than the
copies of RHEL that were currently contracted for service with Red Hat.
Again, this is a “further restriction”. The GPL agreements
give everyone the unfettered right to make and keep as many copies of the
software as they like, and a distributor of GPL’d software may not require
a user to attest that they’ve deleted these legitimate, licensed copies of
third-party-licensed software under the GPL. SFC informed Red Hat’s legal department
of this violation, and we were assured that this additional agreement would no longer
be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a
“tip of the proverbial iceberg”. For years, we have heard from
Red Hat customers who are truly confused. It’s common in the industry to
talk about RHEL “seat licenses”, and many software acquisition
specialists in the industry are not aware of the nuances of the RHEL
business model and do not understand their rights. We remain very
concerned that RHEL salespeople purposely confuse customers to sell more
“seat licenses”. It’s often led us to ask: “If a GPL
violation happens in the woods, and everyone involved doesn’t hear it, how
does anyone know that software rights have indeed been trampled upon in
those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that
are reported to us, and if you’re aware of one, please
do email us at
<[email protected]> immediately. We fear that
be it through incompetence or malice, many RHEL salespeople and business
development professionals may regularly violate GPL and no one knows about
it. That said, the business model as described by IBM’s Red Hat
may well comply with the GPL — it’s just so murky that any tweak to
the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor”
approach — popular in many a shady business deal throughout history. While,
technically speaking, a careful reader of the GPL and the RHEL agreements
understands the bargain they’re making, we suspect most small businesses
just don’t have the FOSS licensing acumen and knowledge to truly understand
that deal.

Why Was an Independent CentOS So Important?

Until Red
Hat’s “aquisition” of CentOS in early 2014, CentOS
provided an excellent counterbalance to the problems with the RHEL
business model. Specifically, CentOS was a community-driven project,
with many volunteers, supported by some involvement from small
businesses, to re-create RHEL releases using the
CCS releases
made for RHEL. Our pre-2014 view was that CentOS was the “canary in
the murky coalmine” of the RHEL business. If CentOS seemed vibrant,
usable, and a viable alternative to RHEL for those who didn’t want to
purchase Red Hat’s updates and services, the community could rest easy.
Even if there were GPL violations by Red Hat on RHEL, CentOS’ vibrancy
assured that such violations were having only a minor negative impact on
the FOSS community around RHEL’s codebase.

Red Hat, however, apparently knew that this vibrant community was cutting
into their profits. Starting in 2013, Red Hat engaged in a series of actions
that increased their grip. First, they “acquired”
CentOS. This was initially couched as a cooperation agreement, but Red Hat
systematically made job offers that key CentOS volunteers couldn’t refuse,
acquired the small businesses who might ultimately build CentOS into a
product, and otherwise integrated CentOS into Red Hat’s own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights
to the CentOS brand as part of the “aquisition”, Red Hat slowly
began to change what CentOS was. CentOS Linux quickly ceased to be a
check-and-balance on RHEL, and just became a testing ground for RHEL.
Then, in 2020, when most of us were distracted by the worst of the COVID-19
pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during
the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely.
IBM’s Red Hat
then used the name “CentOS Stream” to refer to experimental
source packages related to RHEL. These were (and are) not actually the RHEL
source releases — rather, they appear to be primarily a testing
ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago
that RHEL
CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its
CCS publicly
available to everyone. This is a common misconception about GPL’s
requirements. While the details of CCS provisioning vary in the different
versions of the GPL agreements, the general principle is that CCS need to
be provided either (a) along with the binary distributions to those who
receive, or (b) to those who request pursuant to a written offer for
source. In a normal situation, with no mitigating factors, the fact that
a company moved from distributing CCS publicly to everyone to only giving
it to customers who received the binaries already would not raise
concerns.

In this situation, however, this completes what appears to be a
decade-long plan by Red Hat to maximize the level of difficulty of
those in the community who wish to “trust but verify” that RHEL
complies with the GPL agreements. Namely, Red Hat has badly thwarted
efforts by entities such
as Rocky
Linux
and Alma
Linux. These entities are de-facto the intellectual successors to
CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations
sought to build Linux-based distributions that mirrored RHEL
releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM’s Red Hat, the situation has
become increasingly complex and difficult to face. No third party can
effectively monitor RHEL compliance with the GPL agreements, since
customers live in fear of losing their much-needed service contracts.
Red Hat’s legal department
has systematically refused SFC’s requests in recent years to set up some
form of monitoring by SFC. (For example, we asked to review the training
materials and documents that RHEL salespeople are given to convince
customers to buy RHEL, and Red Hat has not been willing to share these
materials with us.) Nevertheless, since SFC serves as the global watchdog for
GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I
personally remember standing with Erik Troan in a Red Hat booth at a USENIX
conference in the late 1990s, and meeting Bob Young around the same time.
Both expressed how much they wanted to build a company that respected,
collaborated with, engaged with, and most of all treated as equals the wide
spectrum of individuals, hobbyists, and small businesses that make the
plurality of the FOSS community. We hope that the
modern Red Hat can find their way back to this mission under IBM’s control.

Posts navigation

← Previous 1 … 1,152 1,153 1,154 … 6,931 Next →

The collective thoughts of the interwebz

Contributors

Rapid7 Cybersecurity Blog

Armed and Dangerous

arp242.net

AWS Architecture Blog

AWS Big Data Blog

AWS Compute Blog

AWS DevOps & Developer Productivity Blog

AWS Messaging Blog

AWS News Blog

AWS Security Blog

Backblaze Blog | Cloud Storage & Cloud Backup

BeardedTinker

Birata.Info

Bivol!

Bozho's tech blog

Bradley M. Kühn's Blog ( bkuhn )

Cloudflare Blog

Crosstalk Solutions

Curious Droid

Darknet – Hacking Tools, Hacker News & Cyber Security

Delian’s Tech blog

Devil’s Advocate Security

digiblur DIY

Errata Security

Explosm.net

fuzzy notepad

Geographics

Grab Tech

Grigor Gatchev – A Weblog

Home Assistant

IBM 360 Model 20 Rescue and Restoration

Joel on Software

KENDOV.COM

LastWeekTonight

laur.ie's blog

lcamtuf’s old blog

Let's Encrypt

LGR

LWN.net

Matt Granger

Matthew Garrett

Monty says

Nebosystems Ltd

Netflix TechBlog – Medium

NTPsec Project Blog

Oglaf! — Comics. Often dirty.

Pid Eins

Prometheus Blog

Raspberry Pi Foundation blog: news, announcements, stories, ideas

Schneier on Security

ServeTheHome

Show Notes

Sprites mods

Talks at Google

Techmoan

Technology Connextras

The Atlantic

The Codeless Code

The History Guy: History Deserves to Be Remembered

The Hook Up

The latest from GitHub’s engineering team – The GitHub Blog

turnoff.us

xkcd.com

Yahoo Engineering

yovko in a nutshell

Zabbix Blog

БЛОГодаря

Блогът на Делян Делчев

Блогът на Юруков

Дневникът на Георги

Дни

Како Сийке, не съм от тях!

Кътчето на Селин

Неосъзнато

татко Крокодил

Тоест

Tags
Advanced (300) AI Amazon Bedrock Amazon EC2 Amazon Redshift Amazon Simple Storage Service (S3) Analytics announcements Architecture artificial intelligence AWS Glue AWS Lambda AWS re:Invent B2Cloud Best practices Cloudflare Workers Cloud Storage comics Customer Solutions cybersecurity Developers Engineering Featured Foundational (100) generative AI intel Intermediate (200) launch networking news Product News Projects research security Security, Identity & Compliance Security Blog serverless squid storage Technical How-to Uncategorized България Водещи Политика общество