Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2025/12/16/rob-reiner-trump-my-brother-also-murdered-my-mother.html

I posted the following on my Fediverse (via Mastodon) account.
I’m reposting the whole seven posts here as written there, but I hope folks
will take a
look at that thread as folks are engaging in conversation over there
that might be worth reading if what I have to say interests you.
(The remainder of the post is the same that can be found in the Fediverse
posts linked throughout.)

I
suppose Fediverse
isn’t the place people are discussing Rob Reiner. But after 36 hours of deliberating whether to say anything, I feel compelled. This thread will be long,but I start w/ most important part:

It’s an “open secret” in the FOSS community that in March 2017 my brother
murdered our mother. About 3k ppl/year in USA have this experience, so it’s
a statistical reality that someone else in FOSS experienced similar. If so,
you’re welcome in my PMs to discuss if you need support… (1/7)

… Traumatic loss due to murder is different than losing your
grandparent/parent of age-related ailments (& is even different than
losing a young person to a disease like cancer). The “a fellow family
member did it” brings permanent surrealism to your daily life. Nothing good
in your life that comes later is ever all that good. I know from direct
experience this is what Rob Reiner’s family now faces. It’s chaos; it
divides families forever: dysfunctional family takes on a new “expert”
level… (2/7)

…as one example: my family was immediately divided about punishment. Some
of my mother’s relatives wanted prosecution to seek death penalty. I knew
that my brother was mentally ill enough that jail or prison *would* get him
killed in a prison dispute eventually,so I met clandestinely w/my brother’s
public defender (during funeral planning!) to get him moved to a criminal
mental health facility instead of a regular prison. If they read this,
it’ll first time my family will find out I did that…(3/7)

…Trump’s political rise (for me) links up: 5 weeks into Trump’s 1ˢᵗ term, my brother murdered my mother.
My (then 33yr-old) brother was severely mentally ill from birth — yet escalated to murder only then. IMO, it wasn’t coincidence. My brother left voicemail approximately 5 hours before the murder stating his intent to murder & described an elaborate political delusion as the impetus.

∃ unintended & dangerous consequences of inflammatory political rhetoric on
the mental ill!…(4/7)

…I’m compelled to speak publicly — for first time ≈10 yrs after the murder —
precisely b/c of Trump’s response.

Trump endorsed the idea that those who oppose him encourage their own
murder from the mentally ill. Indeed, he said that those who oppose him are
*themselves causing* mental illnesses in those around them, & that his
political opponents should *expect* violence from their family members (who
were apparently driven to mental illness from your opposition to
Trump!)… (5/7)

…Trump’s actual words:

Rob Reiner, tortured & struggling,but once…talented movie director &
comedy star, has passed away, together w/ his wife…due to the anger he
caused others through his massive, unyielding, & incurable affliction w/ a
mind crippling disease known as TRUMP DERANGEMENT SYNDROME…He was known to
have driven people CRAZY by his raging obsession of…Trump, w/ his obvious
paranoia reaching new heights as [my] Administration surpassed all goals
and expectations of greatness…

(6/7)

My family became ultra-pro-Trump after my mom’s murder. My mom hated
politics: she was annoyed *both* if I touted my social democratic politics &
if my dad & his family stated their crypto-fascist views.

Every death leaves a hole in a community’s political fabric. 9+ years out,
I’m ostracized from my family b/c I’m anti-Trump.

Trump stated perhaps what my family felt but didn’t say: those who don’t
support Trump are at fault when those who fail to support Trump are
murdered. (7/7)

[ Finally, I want to also quote this one reply I also posted in the same
thread:
I ask everyone, now that I’ve stated this public, that I *know* you’re going to want to search the Internet for it, & you will find a lot. Please, please, keep in mind that the Police Department & others basically lied to the public about some of the facts of the case. I seriously considered suing them for it, but ultimately it wasn’t worth my time. But, please everyone ask me if you are curious about any of the truth of the details of the crime & its aftermath …

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2025/12/16/rob-reiner-trump-my-brother-also-murdered-my-mother.html

I posted the following on my Fediverse (via Mastodon) account.
I’m reposting the whole seven posts here as written there, but I hope folks
will take a
look at that thread as folks are engaging in conversation over there
that might be worth reading if what I have to say interests you.
(The remainder of the post is the same that can be found in the Fediverse
posts linked throughout.)

I
suppose Fediverse
isn’t the place people are discussing Rob Reiner. But after 36 hours of deliberating whether to say anything, I feel compelled. This thread will be long,but I start w/ most important part:

It’s an “open secret” in the FOSS community that in March 2017 my brother
murdered our mother. About 3k ppl/year in USA have this experience, so it’s
a statistical reality that someone else in FOSS experienced similar. If so,
you’re welcome in my PMs to discuss if you need support… (1/7)

… Traumatic loss due to murder is different than losing your
grandparent/parent of age-related ailments (& is even different than
losing a young person to a disease like cancer). The “a fellow family
member did it” brings permanent surrealism to your daily life. Nothing good
in your life that comes later is ever all that good. I know from direct
experience this is what Rob Reiner’s family now faces. It’s chaos; it
divides families forever: dysfunctional family takes on a new “expert”
level… (2/7)

…as one example: my family was immediately divided about punishment. Some
of my mother’s relatives wanted prosecution to seek death penalty. I knew
that my brother was mentally ill enough that jail or prison *would* get him
killed in a prison dispute eventually,so I met clandestinely w/my brother’s
public defender (during funeral planning!) to get him moved to a criminal
mental health facility instead of a regular prison. If they read this,
it’ll first time my family will find out I did that…(3/7)

…Trump’s political rise (for me) links up: 5 weeks into Trump’s 1ˢᵗ term, my brother murdered my mother.
My (then 33yr-old) brother was severely mentally ill from birth — yet escalated to murder only then. IMO, it wasn’t coincidence. My brother left voicemail approximately 5 hours before the murder stating his intent to murder & described an elaborate political delusion as the impetus.

∃ unintended & dangerous consequences of inflammatory political rhetoric on
the mental ill!…(4/7)

…I’m compelled to speak publicly — for first time ≈10 yrs after the murder —
precisely b/c of Trump’s response.

Trump endorsed the idea that those who oppose him encourage their own
murder from the mentally ill. Indeed, he said that those who oppose him are
*themselves causing* mental illnesses in those around them, & that his
political opponents should *expect* violence from their family members (who
were apparently driven to mental illness from your opposition to
Trump!)… (5/7)

…Trump’s actual words:

Rob Reiner, tortured & struggling,but once…talented movie director &
comedy star, has passed away, together w/ his wife…due to the anger he
caused others through his massive, unyielding, & incurable affliction w/ a
mind crippling disease known as TRUMP DERANGEMENT SYNDROME…He was known to
have driven people CRAZY by his raging obsession of…Trump, w/ his obvious
paranoia reaching new heights as [my] Administration surpassed all goals
and expectations of greatness…

(6/7)

My family became ultra-pro-Trump after my mom’s murder. My mom hated
politics: she was annoyed *both* if I touted my social democratic politics &
if my dad & his family stated their crypto-fascist views.

Every death leaves a hole in a community’s political fabric. 9+ years out,
I’m ostracized from my family b/c I’m anti-Trump.

Trump stated perhaps what my family felt but didn’t say: those who don’t
support Trump are at fault when those who fail to support Trump are
murdered. (7/7)

[ Finally, I want to also quote this one reply I also posted in the same
thread:
I ask everyone, now that I’ve stated this public, that I *know* you’re going to want to search the Internet for it, & you will find a lot. Please, please, keep in mind that the Police Department & others basically lied to the public about some of the facts of the case. I seriously considered suing them for it, but ultimately it wasn’t worth my time. But, please everyone ask me if you are curious about any of the truth of the details of the crime & its aftermath …

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2025/12/16/rob-reiner-trump-my-brother-also-murdered-my-mother.html

I posted the following on my Fediverse (via Mastodon) account.
I’m reposting the whole seven posts here as written there, but I hope folks
will take a
look at that thread as folks are engaging in conversation over there
that might be worth reading if what I have to say interests you.
(The remainder of the post is the same that can be found in the Fediverse
posts linked throughout.)

I
suppose Fediverse
isn’t the place people are discussing Rob Reiner. But after 36 hours of deliberating whether to say anything, I feel compelled. This thread will be long,but I start w/ most important part:

It’s an “open secret” in the FOSS community that in March 2017 my brother
murdered our mother. About 3k ppl/year in USA have this experience, so it’s
a statistical reality that someone else in FOSS experienced similar. If so,
you’re welcome in my PMs to discuss if you need support… (1/7)

… Traumatic loss due to murder is different than losing your
grandparent/parent of age-related ailments (& is even different than
losing a young person to a disease like cancer). The “a fellow family
member did it” brings permanent surrealism to your daily life. Nothing good
in your life that comes later is ever all that good. I know from direct
experience this is what Rob Reiner’s family now faces. It’s chaos; it
divides families forever: dysfunctional family takes on a new “expert”
level… (2/7)

…as one example: my family was immediately divided about punishment. Some
of my mother’s relatives wanted prosecution to seek death penalty. I knew
that my brother was mentally ill enough that jail or prison *would* get him
killed in a prison dispute eventually,so I met clandestinely w/my brother’s
public defender (during funeral planning!) to get him moved to a criminal
mental health facility instead of a regular prison. If they read this,
it’ll first time my family will find out I did that…(3/7)

…Trump’s political rise (for me) links up: 5 weeks into Trump’s 1ˢᵗ term, my brother murdered my mother.
My (then 33yr-old) brother was severely mentally ill from birth — yet escalated to murder only then. IMO, it wasn’t coincidence. My brother left voicemail approximately 5 hours before the murder stating his intent to murder & described an elaborate political delusion as the impetus.

∃ unintended & dangerous consequences of inflammatory political rhetoric on
the mental ill!…(4/7)

…I’m compelled to speak publicly — for first time ≈10 yrs after the murder —
precisely b/c of Trump’s response.

Trump endorsed the idea that those who oppose him encourage their own
murder from the mentally ill. Indeed, he said that those who oppose him are
*themselves causing* mental illnesses in those around them, & that his
political opponents should *expect* violence from their family members (who
were apparently driven to mental illness from your opposition to
Trump!)… (5/7)

…Trump’s actual words:

Rob Reiner, tortured & struggling,but once…talented movie director &
comedy star, has passed away, together w/ his wife…due to the anger he
caused others through his massive, unyielding, & incurable affliction w/ a
mind crippling disease known as TRUMP DERANGEMENT SYNDROME…He was known to
have driven people CRAZY by his raging obsession of…Trump, w/ his obvious
paranoia reaching new heights as [my] Administration surpassed all goals
and expectations of greatness…

(6/7)

My family became ultra-pro-Trump after my mom’s murder. My mom hated
politics: she was annoyed *both* if I touted my social democratic politics &
if my dad & his family stated their crypto-fascist views.

Every death leaves a hole in a community’s political fabric. 9+ years out,
I’m ostracized from my family b/c I’m anti-Trump.

Trump stated perhaps what my family felt but didn’t say: those who don’t
support Trump are at fault when those who fail to support Trump are
murdered. (7/7)

[ Finally, I want to also quote this one reply I also posted in the same
thread:
I ask everyone, now that I’ve stated this public, that I *know* you’re going to want to search the Internet for it, & you will find a lot. Please, please, keep in mind that the Police Department & others basically lied to the public about some of the facts of the case. I seriously considered suing them for it, but ultimately it wasn’t worth my time. But, please everyone ask me if you are curious about any of the truth of the details of the crime & its aftermath …

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2025/12/16/rob-reiner-trump-my-brother-also-murdered-my-mother.html

I posted the following on my Fediverse (via Mastodon) account.
I’m reposting the whole seven posts here as written there, but I hope folks
will take a
look at that thread as folks are engaging in conversation over there
that might be worth reading if what I have to say interests you.
(The remainder of the post is the same that can be found in the Fediverse
posts linked throughout.)

I
suppose Fediverse
isn’t the place people are discussing Rob Reiner. But after 36 hours of deliberating whether to say anything, I feel compelled. This thread will be long,but I start w/ most important part:

It’s an “open secret” in the FOSS community that in March 2017 my brother
murdered our mother. About 3k ppl/year in USA have this experience, so it’s
a statistical reality that someone else in FOSS experienced similar. If so,
you’re welcome in my PMs to discuss if you need support… (1/7)

… Traumatic loss due to murder is different than losing your
grandparent/parent of age-related ailments (& is even different than
losing a young person to a disease like cancer). The “a fellow family
member did it” brings permanent surrealism to your daily life. Nothing good
in your life that comes later is ever all that good. I know from direct
experience this is what Rob Reiner’s family now faces. It’s chaos; it
divides families forever: dysfunctional family takes on a new “expert”
level… (2/7)

…as one example: my family was immediately divided about punishment. Some
of my mother’s relatives wanted prosecution to seek death penalty. I knew
that my brother was mentally ill enough that jail or prison *would* get him
killed in a prison dispute eventually,so I met clandestinely w/my brother’s
public defender (during funeral planning!) to get him moved to a criminal
mental health facility instead of a regular prison. If they read this,
it’ll first time my family will find out I did that…(3/7)

…Trump’s political rise (for me) links up: 5 weeks into Trump’s 1ˢᵗ term, my brother murdered my mother.
My (then 33yr-old) brother was severely mentally ill from birth — yet escalated to murder only then. IMO, it wasn’t coincidence. My brother left voicemail approximately 5 hours before the murder stating his intent to murder & described an elaborate political delusion as the impetus.

∃ unintended & dangerous consequences of inflammatory political rhetoric on
the mental ill!…(4/7)

…I’m compelled to speak publicly — for first time ≈10 yrs after the murder —
precisely b/c of Trump’s response.

Trump endorsed the idea that those who oppose him encourage their own
murder from the mentally ill. Indeed, he said that those who oppose him are
*themselves causing* mental illnesses in those around them, & that his
political opponents should *expect* violence from their family members (who
were apparently driven to mental illness from your opposition to
Trump!)… (5/7)

…Trump’s actual words:

Rob Reiner, tortured & struggling,but once…talented movie director &
comedy star, has passed away, together w/ his wife…due to the anger he
caused others through his massive, unyielding, & incurable affliction w/ a
mind crippling disease known as TRUMP DERANGEMENT SYNDROME…He was known to
have driven people CRAZY by his raging obsession of…Trump, w/ his obvious
paranoia reaching new heights as [my] Administration surpassed all goals
and expectations of greatness…

(6/7)

My family became ultra-pro-Trump after my mom’s murder. My mom hated
politics: she was annoyed *both* if I touted my social democratic politics &
if my dad & his family stated their crypto-fascist views.

Every death leaves a hole in a community’s political fabric. 9+ years out,
I’m ostracized from my family b/c I’m anti-Trump.

Trump stated perhaps what my family felt but didn’t say: those who don’t
support Trump are at fault when those who fail to support Trump are
murdered. (7/7)

[ Finally, I want to also quote this one reply I also posted in the same
thread:
I ask everyone, now that I’ve stated this public, that I *know* you’re going to want to search the Internet for it, & you will find a lot. Please, please, keep in mind that the Police Department & others basically lied to the public about some of the facts of the case. I seriously considered suing them for it, but ultimately it wasn’t worth my time. But, please everyone ask me if you are curious about any of the truth of the details of the crime & its aftermath …

Post Syndicated from Matt Granger original https://www.youtube.com/watch?v=dmKrOrXD4Ho

Post Syndicated from Arve Knudsen original https://prometheus.io/blog/2025/12/16/introducing-info-function/

Enriching metrics with metadata labels can be surprisingly tricky in Prometheus, even if you’re a PromQL wiz!
The PromQL join query traditionally used for this is inherently quite complex because it has to specify the labels to join on, the info metric to join with, and the labels to enrich with.
The new, still experimental info() function, promises a simpler way, making label enrichment as simple as wrapping your query in a single function call.

In Prometheus 3.0, we introduced the info() function, a powerful new way to enrich your time series with labels from info metrics.
What’s special about info() versus the traditional join query technique is that it relieves you from having to specify identifying labels, which info metric(s) to join with, and the (“data” or “non-identifying”) labels to enrich with.
Note that “identifying labels” in this particular context refers to the set of labels that identify the info metrics in question, and are shared with associated non-info metrics.
They are the labels you would join on in a Prometheus join query.
Conceptually, they can be compared to foreign keys in relational databases.

Beyond the main functionality, info() also solves a subtle yet critical problem that has plagued join queries for years: The “churn problem” that causes queries to fail when non-identifying info metric labels change, combined with missing staleness marking (as is the case with OTLP ingestion).

Whether you’re working with OpenTelemetry resource attributes, Kubernetes labels, or any other metadata, the info() function makes your PromQL queries cleaner, more reliable, and easier to understand.

The Problem: Complex Joins and The Churn Problem

Let us start by looking at what we have had to do until now.
Imagine you’re monitoring HTTP request durations via OpenTelemetry and want to break them down by Kubernetes cluster.
You push your metrics to Prometheus’ OTLP endpoint.
Your metrics have job and instance labels, but the cluster name lives in a separate target_info metric, as the k8s_cluster_name label.
Here’s what the traditional approach looks like:

sum by (http_status_code, k8s_cluster_name) (
    rate(http_server_request_duration_seconds_count[2m])
  * on (job, instance) group_left (k8s_cluster_name)
    target_info
)

While this works, there are several issues:

1. Complexity: You need to know:

• Which info metric contains your labels (target_info)
• Which labels are the “identifying” labels to join on (job, instance)
• Which data labels you want to add (k8s_cluster_name)
• The proper PromQL join syntax (on, group_left)

This requires expert-level PromQL knowledge and makes queries harder to read and maintain.

2. The Churn Problem (The Critical Issue):

Here’s the subtle but serious problem: What happens when an OTel resource attribute changes in a Kubernetes container, while the identifying resource attributes stay the same?
An example could be the resource attribute k8s.pod.labels.app.kubernetes.io/version.
Then the corresponding target_info label k8s_pod_labels_app_kubernetes_io_version changes, and Prometheus sees a completely new target_info time series.

As the OTLP endpoint doesn’t mark the old target_info series as stale, both the old and new series can exist simultaneously for up to 5 minutes (the default lookback delta).
During this overlap period, your join query finds two distinct matching target_info time series and fails with a “many-to-many matching” error.

This could in practice mean your dashboards break and your alerts stop firing when infrastructure changes are happening, perhaps precisely when you would need visibility the most.

The Info Function Presents a Solution

The previous join query can be converted to use the info function as follows:

sum by (http_status_code, k8s_cluster_name) (
  info(rate(http_server_request_duration_seconds_count[2m]))
)

Much more comprehensible, isn’t it?
As regards solving the churn problem, the real magic happens under the hood: info() automatically selects the time series with the latest sample, eliminating churn-related join failures entirely.
Note that this call to info() returns all data labels from target_info, but it doesn’t matter because we aggregate them away with sum.

Basic Syntax

info(v instant-vector, [data-label-selector instant-vector])

• v: The instant vector to enrich with metadata labels
• data-label-selector (optional): Label matchers in curly braces to filter which labels to include

In its most basic form, omitting the second parameter, info() adds all data labels from target_info:

info(rate(http_server_request_duration_seconds_count[2m]))

Through the second parameter on the other hand, you can control which data labels to include from target_info:

info(
  rate(http_server_request_duration_seconds_count[2m]),
  {k8s_cluster_name=~".+"}
)

In the example above, info() includes the k8s_cluster_name data label from target_info.
Because the selector matches any non-empty string, it will include any k8s_cluster_name label value.

It’s also possible to filter which k8s_cluster_name label values to include:

info(
  rate(http_server_request_duration_seconds_count[2m]),
  {k8s_cluster_name="us-east-0"}
)

Selecting Different Info Metrics

By default, info() uses the target_info metric.
However, you can select different info metrics (like build_info or node_uname_info) by including a __name__ matcher in the data-label-selector:

# Use build_info instead of target_info
info(up, {__name__="build_info"})

# Use multiple info metrics (combines labels from both)
info(up, {__name__=~"(target|build)_info"})

# Select build_info and only include the version label
info(up, {__name__="build_info", version=~".+"})

Note: The current implementation always uses job and instance as the identifying labels for joining, regardless of which info metric you select.
This works well for most standard info metrics but may have limitations with custom info metrics that use different identifying labels.
An example of an info metric that has different identifying labels than job and instance is kube_pod_labels, its identifying labels are instead: namespace and pod.
The intention is that info() in the future knows which metrics in the TSDB are info metrics and automatically uses all of them, unless the selection is explicitly restricted by a name matcher like the above, and which are the identifying labels for each info metric.

Real-World Use Cases

OpenTelemetry Integration

The primary driver for the info() function is OpenTelemetry (OTel) integration.
When using Prometheus as an OTel backend, resource attributes (metadata about the metrics producer) are automatically converted to the target_info metric:

• service.instance.id → instance label
• service.name → job label
• service.namespace → prefixed to job (i.e., <namespace>/<service.name>)
• All other resource attributes → data labels on target_info

This means that, so long as at least either the service.instance.id or the service.name resource attribute is included, every OTel metric you send to Prometheus over OTLP can be enriched with resource attributes using info():

# Add all OTel resource attributes
info(rate(http_server_request_duration_seconds_sum[5m]))

# Add only specific attributes
info(
  rate(http_server_request_duration_seconds_sum[5m]),
  {k8s_cluster_name=~".+", k8s_namespace_name=~".+", k8s_pod_name=~".+"}
)

Build Information

Enrich your metrics with build-time information:

# Add version and branch information to request rates
sum by (job, http_status_code, version, branch) (
  info(
    rate(http_server_request_duration_seconds_count[2m]),
    {__name__="build_info"}
  )
)

Filter on Producer Version

Pick only metrics from certain producer versions:

sum by (job, http_status_code, version) (
  info(
    rate(http_server_request_duration_seconds_count[2m]),
    {__name__="build_info", version=~"2\\..+"}
  )
)

Before and After: Side-by-Side Comparison

Let’s see how the info() function simplifies real queries:

Example 1: OpenTelemetry Resource Attribute Enrichment

Traditional approach:

sum by (http_status_code, k8s_cluster_name, k8s_namespace_name, k8s_container_name) (
    rate(http_server_request_duration_seconds_count[2m])
  * on (job, instance) group_left (k8s_cluster_name, k8s_namespace_name, k8s_container_name)
    target_info
)

With info():

sum by (http_status_code, k8s_cluster_name, k8s_namespace_name, k8s_container_name) (
  info(rate(http_server_request_duration_seconds_count[2m]))
)

The intent is much clearer with info: We’re enriching http_server_request_duration_seconds_count with Kubernetes related OpenTelemetry resource attributes.

Example 2: Filtering by Label Value

Traditional approach:

sum by (http_status_code, k8s_cluster_name) (
    rate(http_server_request_duration_seconds_count[2m])
  * on (job, instance) group_left (k8s_cluster_name)
    target_info{k8s_cluster_name=~"us-.*"}
)

With info():

sum by (http_status_code, k8s_cluster_name) (
  info(
    rate(http_server_request_duration_seconds_count[2m]),
    {k8s_cluster_name=~"us-.*"}
  )
)

Here we filter to only include metrics from clusters in the US (which names start with us-). The info() version integrates the filter naturally into the data-label-selector.

Technical Benefits

Beyond the fundamental UX benefits, the info() function provides several technical advantages:

1. Automatic Churn Handling

As previously mentioned, info() automatically picks the matching info time series with the latest sample when multiple versions exist.
This eliminates the “many-to-many matching” errors that plague traditional join queries during churn.

How it works: When non-identifying info metric labels change (e.g., a pod is re-created), there’s a brief period where both old and new series might exist.
The info() function simply selects whichever has the most recent sample, ensuring your queries keep working.

2. Better Performance

The info() function is more efficient than traditional joins:

• Only selects matching info series
• Avoids unnecessary label matching operations
• Optimized query execution path

Getting Started

The info() function is experimental and must be enabled via a feature flag:

prometheus --enable-feature=promql-experimental-functions

Once enabled, you can start using it immediately.

Current Limitations and Future Plans

The current implementation is an MVP (Minimum Viable Product) designed to validate the approach and gather user feedback.
The implementation has some intentional limitations:

Current Constraints

1. Default info metric: Only considers target_info by default

   • Workaround: You can use __name__ matchers like {__name__=~"(target|build)_info"} in the data-label-selector, though this still assumes job and instance as identifying labels

2. Fixed identifying labels: Always assumes job and instance are the identifying labels for joining

   • This unfortunately makes info() unsuitable for certain scenarios, e.g. including data labels from kube_pod_labels, but it’s a problem we want to solve in the future

Future Development

These limitations are meant to be temporary.
The experimental status allows us to:

• Gather real-world usage feedback
• Understand which use cases matter the most
• Iterate on the design before committing to a final API

A future version of the info() function should:

• Consider all info metrics by default (not just target_info)
• Automatically understand identifying labels based on info metric metadata

Important: Because this is an experimental feature, the behavior may change in future Prometheus versions, or the function could potentially be removed from PromQL entirely based on user feedback.

Giving Feedback

Your feedback will directly shape the future of this feature and help us determine whether it should become a permanent part of PromQL.
Feedback may be provided e.g. through our community connections or by opening a Prometheus issue.

We encourage you to try the info() function and share your feedback:

• What use cases does it solve for you?
• What additional functionality would you like to see?
• How could the API be improved?
• Do you see improved performance?

Conclusion

The experimental info() function represents a significant step forward in making PromQL more accessible and reliable.
By simplifying metadata label enrichment and automatically handling the churn problem, it removes two major pain points for Prometheus users, especially those adopting OpenTelemetry.

To learn more:

• PromQL functions documentation
• OpenTelemetry guide (includes detailed info() usage)
• Feature proposal

Please feel welcome to share your thoughts with the Prometheus community on GitHub Discussions or get in touch with us on the CNCF Slack #prometheus channel.

Happy querying!

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/how-temporal-powers-reliable-cloud-operations-at-netflix-73c69ccb5953

By Jacob Meyers and Rob Zienert

Temporal is a Durable Execution platform which allows you to write code “as if failures don’t exist”. It’s become increasingly critical to Netflix since its initial adoption in 2021, with users ranging from the operators of our Open Connect global CDN to our Live reliability teams now depending on Temporal to operate their business-critical services. In this post, I’ll give a high-level overview of what Temporal offers users, the problems we were experiencing operating Spinnaker that motivated its initial adoption at Netflix, and how Temporal helped us reduce the number of transient deployment failures at Netflix from 4% to 0.0001%.

A Crash Course on (some of) Spinnaker

Spinnaker is a multi-cloud continuous delivery platform that powers the vast majority of Netflix’s software deployments. It’s composed of several (mostly nautical themed) microservices. Let’s double-click on two in particular to understand the problems we were facing that led us to adopting Temporal.

In case you’re completely new to Spinnaker, Spinnaker’s fundamental tool for deployments is the Pipeline. A Pipeline is composed of a sequence of steps called Stages, which themselves can be decomposed into one or more Tasks, or other Stages. An example deployment pipeline for a production service may consist of these stages: Find Image -> Run Smoke Tests -> Run Canary -> Deploy to us-east-2 -> Wait -> Deploy to us-east-1.

Pipeline configuration is extremely flexible. You can have Stages run completely serially, one after another, or you can have a mix of concurrent and serial Stages. Stages can also be executed conditionally based on the result of previous stages. This brings us to our first Spinnaker service: Orca. Orca is the orca-stration engine of Spinnaker. It’s responsible for managing the execution of the Stages and Tasks that a Pipeline unrolls into and coordinating with other Spinnaker services to actually execute them.

One of those collaborating services is called Clouddriver. In the example Pipeline above, some of the Stages will require interfacing with cloud infrastructure. For example, the canary deployment involves creating ephemeral hosts to run an experiment, and a full deployment of a new version of the service may involve spinning up new servers and then tearing down the old ones. We call these sorts of operations that mutate cloud infrastructure Cloud Operations. Clouddriver’s job is to decompose and execute Cloud Operations sent to it by Orca as part of a deployment. Cloud Operations sent from Orca to Clouddriver are relatively high level (for example: createServerGroup), so Clouddriver understands how to translate these into lower-level cloud provider API calls.

Pain points in the interaction between Orca and Clouddriver and the implementation details of Cloud Operation execution in Clouddriver are what led us to look for new solutions and ultimately migrate to Temporal, so we’ll next look at the anatomy of a Cloud Operation. Cloud Operations in the OSS version of Spinnaker still work as described below, so motivated readers can follow along in source code, however our migration to Temporal is entirely closed-source following a fork from OSS in 2020 to allow Netflix to make larger pivots to the product such as this one.

The Original Cloud Operation Flow

A Cloud Operation’s execution goes something like this:

1. Orca, in orchestrating a Pipeline execution, decides a particular Cloud Operation needs to be performed. It sends a POST request to Clouddriver’s /ops endpoint with an untyped bag-of-fields.
2. Clouddriver attempts to resolve the operation Orca sent into a set of AtomicOperation s— internal operations that only Clouddriver understands.
3. If the payload was valid and Clouddriver successfully resolved the operation, it will immediately return a Task ID to Orca.
4. Orca will immediately begin polling Clouddriver’s GET /task/<id> endpoint to keep track of the status of the Cloud Operation.
5. Asynchronously, Clouddriver begins executing AtomicOperations using its own internal orchestration engine. Ultimately, the AtomicOperations resolve into cloud provider API calls. As the Cloud Operation progresses, Clouddriver updates an internal state store to surface progress to Orca.
6. Eventually, if all went well, Clouddriver will mark the Cloud Operation complete, which eventually surfaces to Orca in its polling. Orca considers the Cloud Operation finished, and the deployment can progress.

This works well enough on the happy path, but veer off the happy path and dragons begin to emerge:

1. Clouddriver has its own internal orchestration system independent of Orca to allow Orca to query the progress of Cloud Operation. This is largely undifferentiated lifting relative to Clouddriver’s goal of actuating cloud infrastructure changes, and ultimately adds complexity and surface area for bugs to the application. Additionally, Orca is tightly coupled to Clouddriver’s orchestration system — it must understand how to poll Clouddriver, interpret the status, and handle errors returned by Clouddriver.
2. Distributed systems are messy — networks and external services are unreliable. While executing a Cloud Operation, Clouddriver could experience transient network issues, or the cloud provider it’s attempting to call into may be having an outage, or any number of issues in between. Despite all of this, Clouddriver must be as reliable as reasonably possible as a core platform service. To deal with this shape of issue, Clouddriver internally evolved complex retry logic, further adding cognitive complexity to the system.
3. Remember how a Cloud Operation gets decomposed by Clouddriver into AtomicOperations? Sometimes, if there’s a failure in the middle of a Cloud Operation, we need to be able to roll back what was done in AtomicOperations prior to the failure. This led to a homegrown Saga framework being implemented inside Clouddriver. While this did result in a big step forward in reliability of Cloud Operations facing transient failures because the Saga framework also allowed replaying partially-failed Cloud Operations, it added yet more undifferentiated lifting inside the service.
4. The task state kept by Clouddriver was instance-local. In other words, if the Clouddriver instance carrying out a Cloud Operation crashed, that Cloud Operation state was lost, and Orca would eventually time out polling for the task status. The Saga implementation mentioned above mitigated this for certain operations, but was not widely adopted across all cloud providers supported by Spinnaker.

We introduced a lot of incidental complexity into Clouddriver in an effort to keep Cloud Operation execution reliable, and despite all this deployments still failed around 4% of the time due to transient Cloud Operation failures.

Now, I can already hear you saying: “So what? Can’t people re-try their deployments if they fail?” While true, some pipelines take days to complete for complex deployments, and a failed Cloud Operation mid-way through requires re-running the whole thing. This was detrimental to engineering productivity at Netflix in a non-trivial way. Rather than continue trying to build a faster horse, we began to look elsewhere for our reliable orchestration requirements, which is where Temporal comes in.

Temporal: Basic Concepts

Temporal is an open source product that offers a durable execution platform for your applications. Durable execution means that the platform will ensure your programs run to completion despite adverse conditions. With Temporal, you organize your business logic into Workflows, which are a deterministic series of steps. The steps inside of Workflows are called Activities, which is where you encapsulate all your non-deterministic logic that needs to happen in the course of executing your Workflows. As your Workflows execute in processes called Workers, the Temporal server durably stores their execution state so that in the event of failures your Workflows can be retried or even migrated to a different Worker. This makes Workflows incredibly resilient to the sorts of transient failures Clouddriver was susceptible to. Here’s a simple example Workflow in Java that runs an Activity to send an email once every 30 days:

@WorkflowInterface
public interface SleepForDaysWorkflow {
    @WorkflowMethod
    void run();
}

public class SleepForDaysWorkflowImpl implements SleepForDaysWorkflow {

    private final SendEmailActivities emailActivities =
            Workflow.newActivityStub(
                    SendEmailActivities.class,
                    ActivityOptions.newBuilder()
                            .setStartToCloseTimeout(Duration.ofSeconds(10))
                            .build());

    @Override
    public void run() {
        while (true) {
            // Activities already carry retries/timeouts via options.
            emailActivities.sendEmail();

            // Pause the workflow for 30 days before sending the next email.
            Workflow.sleep(Duration.ofDays(30));
        }
    }
}

@ActivityInterface
public interface SendEmailActivities {
    void sendEmail();
}

There’s some interesting things to note about this Workflow:

1. Workflows and Activities are just code, so you can test them using the same techniques and processes as the rest of your codebase.
2. Activities are automatically retried by Temporal with configurable exponential backoff.
3. Temporal manages all the execution state of the Workflow, including timers (like the one used by Workflow.sleep). If the Worker executing this workflow were to have its power cable unplugged, Temporal would ensure another Worker continues to execute it (even during the 30 day sleep).
4. Workflow sleeps are not compute-intensive, and they don’t tie up the process.

You might already begin to see how Temporal solves a lot of the problems we had with Clouddriver. Ultimately, we decided to pull the trigger on migrating Cloud Operation execution to Temporal.

Cloud Operations with Temporal

Today, we execute Cloud Operations as Temporal workflows. Here’s what that looks like.

1. Orca, using a Temporal client, sends a request to Temporal to execute an UntypedCloudOperationRunner Workflow. The contract of the Workflow looks something like this:

@WorkflowInterface
interface UntypedCloudOperationRunner {
  /**
   * Runs a cloud operation given an untyped payload.
   *
   * WorkflowResult is a thin wrapper around OutputType providing a standard contract for
   * clients to determine if the CloudOperation was successful and fetching any errors.
   */
  @WorkflowMethod
  fun <OutputType : CloudOperationOutput> run(stageContext: Map<String, Any?>, operationType: String): WorkflowResult<OutputType>
}

2. The Clouddriver Temporal worker is constantly polling Temporal for work. A worker will eventually see a task for an UntypedCloudOperationRunner Workflow and start executing it.

3. Similar to before with resolution into AtomicOperations, Clouddriver does some pre-processing of the bag-of-fields in stageContext and resolves it to a strongly typed implementation of the CloudOperation Workflow interface based on the operationType input and the stageContext:

interface CloudOperation<I : CloudOperationInput, O : CloudOperationOutput> {
  @WorkflowMethod
  fun operate(input: I, credentials: AccountCredentials<out Any>): O
}

4. Clouddriver starts a Child Workflow execution of the CloudOperation implementation it resolved. The child workflow will execute Activities which handle the actual cloud provider API calls to mutate infrastructure.

5. Orca uses its Temporal Client to await completion of the UntypedCloudOperationRunner Workflow. Once it’s complete, Temporal notifies the client and sends the result and Orca can continue progressing the deployment.

Results and Lessons Learned from the Migration

A shiny new architecture is great, but equally important is the non-glamorous work of refactoring legacy systems to fit the new architecture. How did we integrate Temporal into critical dependencies of all Netflix engineers transparently?

The answer, of course, is a combination of abstraction and dynamic configuration. We built a CloudOperationRunner interface in Orca to encapsulate whether the Cloud Operation was being executed via the legacy path or Temporal. At runtime, Fast Properties (Netflix’s dynamic configuration system) determined which path a stage that needed to execute a Cloud Operation would take. We could set these properties quite granularly — by Stage type, cloud provider account, Spinnaker application, Cloud Operation type (createServerGroup), and cloud provider (either AWS or Titus in our case). The Spinnaker services themselves were the first to be deployed using Temporal, and within two quarters, all applications at Netflix were onboarded.

Impact

What did we have to show for it all? With Temporal as the orchestration engine for Cloud Operations, the percentage of deployments that failed due to transient Cloud Operation failures dropped from 4% to 0.0001%. For those keeping track at home, that’s a four and a half order of magnitude reduction. Virtually eliminating this failure mode for deployments was a huge win for developer productivity, especially for teams with long and complex deployment pipelines.

Beyond the improvement in deployment success metrics, we saw a number of other benefits:

1. Orca no longer needs to directly communicate with Clouddriver to start Cloud Operations or poll their status with Temporal as the intermediary. The services are less coupled, which is a win for maintainability.
2. Speaking of maintainability, with Temporal doing the heavy lifting of orchestration and retries inside of Clouddriver, we got to remove a lot of the homegrown logic we’d built up over the years for the same purpose.
3. Since Temporal manages execution state, Clouddriver instances became stateless and Cloud Operation execution can bounce between instances with impunity. We can treat Clouddriver instances more like cattle and enable things like Chaos Monkey for the service which we were previously prevented from doing.
4. Migrating Cloud Operation steps into Activities was a forcing function to re-write the logic to be idempotent. Since Temporal retries activities by default, it’s generally recommended they be idempotent. This alone fixed a number of issues that existed previously when operations were retried in Clouddriver.
5. We set the retry timeout for Activities in Clouddriver to be two hours by default. This gives us a long leash to fix-forward or rollback Clouddriver if we introduce a regression before customer deployments fail — to them, it might just look like a deployment is taking longer than usual.
6. Cloud Operations are much easier to introspect than before. Temporal ships with a great UI to help visualize Workflow and Activity executions, which is a huge boon for debugging live Workflows executing in production. The Temporal SDKs and server also emit a lot of useful metrics.

Lessons Learned

With the benefit of hindsight, there are also some lessons we can share from this migration:

1. Avoid unnecessary Child Workflows: Structuring Cloud Operations as an UntypedCloudOperationRunner Workflow that starts Child Workflows to actually execute the Cloud Operation’s logic was unnecessary and the indirection made troubleshooting more difficult. There are situations where Child Workflows are appropriate, but in this case we were using them as a tool for code organization, which is generally unnecessary. We could’ve achieved the same effect with class composition in the top-level parent Workflow.

2. Use single argument objects: At first, we structured Workflow and Activity functions with variable arguments, much as you’d write normal functions. This can be problematic for Temporal because of Temporal’s determinism constraints. Adding or removing an argument from a function signature is not a backward-compatible change, and doing so can break long-running workflows — and it’s not immediately obvious in code review your change is problematic. The preferred pattern is to use a single serializable class to host all your arguments for Workflows and Activities — these can be more freely changed without breaking determinism.

3. Separate business failures from workflow failures: We like the pattern of the WorkflowResult type that UntypedCloudOperationRunner returns in the interface above. It allows us to communicate business process failures without failing the Workflow itself and have more overall nuance in error handling. This is a pattern we’ve carried over to other Workflows we’ve implemented since.

Temporal at Netflix Today

Temporal adoption has skyrocketed at Netflix since its initial introduction for Spinnaker. Today, we have hundreds of use cases, and we’ve seen adoption double in the last year with no signs of slowing down.

One major difference between initial adoption and today is that Netflix migrated from an on-prem Temporal deployment to using Temporal Cloud, which is Temporal’s SaaS offering of the Temporal server. This has let us scale Temporal adoption while running a lean team. We’ve also built up a robust internal platform around Temporal Cloud to integrate with Netflix’s internal ecosystem and make onboarding for our developers as easy as possible. Stay tuned for a future post digging into more specifics of our Netflix Temporal platform.

Acknowledgement

We all stand on the shoulders of giants in software. I want to call out that I’m retelling the work of my two stunning colleagues Chris Smalley and Rob Zienert in this post, who were the two aforementioned engineers who introduced Temporal and carried out the migration.

---

How Temporal Powers Reliable Cloud Operations at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Post Syndicated from jzb original https://lwn.net/Articles/1049886/

Version
8.16.0 of the calibre
ebook-management software, released on December 4, includes a
“Discuss with AI” feature that can be used to query various AI/LLM
services or local models about books, and ask for recommendations on
what to read next. The feature has sparked discussion among human
users of calibre as well, and more than a few are upset about the
intrusion of AI into the software. After much pushback, it looks as
though users will get the ability to hide the feature from calibre’s user
interface, but LLM-driven features are here to stay and more will
likely be added over time.