Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Teaching about AI in schools: Take part in our Research and Educator Community Symposium

Post Syndicated from Jane Waite original https://www.raspberrypi.org/blog/teaching-about-ai-in-schools-research-and-educator-community-symposium/

Worldwide, the use of generative AI systems and related technologies is transforming our lives. From marketing and social media to education and industry, these technologies are being used everywhere, even if it isn’t obvious. Yet, despite the growing availability and use of generative AI tools, governments are still working out how and when to regulate such technologies to ensure they don’t cause unforeseen negative consequences.

How, then, do we equip our young people to deal with the opportunities and challenges that they are faced with from generative AI applications and associated systems? Teaching them about AI technologies seems an important first step. But what should we teach, when, and how?

A teacher aids children in the classroom

Researching AI curriculum design

The researchers at the Raspberry Pi Foundation have been looking at research that will help inform curriculum design and resource development to teach about AI in school. As part of this work, a number of research themes have been established, which we would like to explore with educators at a face-to-face symposium. 

These research themes include the SEAME model, a simple way to analyse learning experiences about AI technology, as well as anthropomorphisation and how this might influence the formation of mental models about AI products. These research themes have become the cornerstone of the Experience AI resources we’ve co-developed with Google DeepMind. We will be using these materials to exemplify how the research themes can be used in practice as we review the recently published UNESCO AI competencies.

A group of educators at a workshop.

Most importantly, we will also review how we can help teachers and learners move from a rule-based view of problem solving to a data-driven view, from computational thinking 1.0 to computational thinking 2.0.

A call for teacher input on the AI curriculum

Over ten years ago, teachers in England experienced a large-scale change in what they needed to teach in computing lessons when programming was more formally added to the curriculum. As we enter a similar period of change — this time to introduce teaching about AI technologies — we want to hear from teachers as we collectively start to rethink our subject and curricula. 

We think it is imperative that educators’ voices are heard as we reimagine computer science and add data-driven technologies into an already densely packed learning context. 

Educators at a workshop.

Join our Research and Educator Community Symposium

On Saturday, 1 February 2025, we are running a Research and Educator Community Symposium in collaboration with the Raspberry Pi Computing Education Research Centre. 

In this symposium, we will bring together UK educators and researchers to review research themes, competency frameworks, and early international AI curricula and to reflect on how to advance approaches to teaching about AI. This will be a practical day of collaboration to produce suggested key concepts and pedagogical approaches and highlight research needs. 

Educators and researchers at an event.

This symposium focuses on teaching about AI technologies, so we will not be looking at which AI tools might be used in general teaching and learning or how they may change teacher productivity. 

It is vitally important for young people to learn how to use AI technologies in their daily lives so they can become discerning consumers of AI applications. But how should we teach them? Please help us start to consider the best approach by signing up for our Research and Educator Community Symposium by 9 December 2024.

Information at a glance

When:  Saturday, 1 February 2025 (10am to 5pm) 

Where: Raspberry Pi Foundation Offices, Cambridge

Who: If you have started teaching about AI, are creating related resources, are providing professional development about AI technologies, or if you are planning to do so, please apply to attend our symposium. Travel funding is available for teachers in England.

Apply by 9 December 2024

Please note we expect to be oversubscribed, so book early and tell us about why you are interested in taking part. We will notify all applicants of the outcome of their application by 11 December.

The post Teaching about AI in schools: Take part in our Research and Educator Community Symposium appeared first on Raspberry Pi Foundation.

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

Open Source AI Definition Erodes the Meaning of “Open Source”

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2024/10/31/open-source-ai-osaid-osi.html

[ This is
a crosspost
from my professional blog at Software Freedom Conservancy
(SFC). I encourage you
to use
that copy of the post as the canonical linkage for this essay — I
crossposted here merely for posterity and to reach a wider
audience. ]

This week, the Open Source Initiative (OSI) made their new Open
Source Artificial Intelligence Definition (OSAID) official with its 1.0 release. With this
announcement, we have reached the moment that software freedom advocates have
feared for decades: the definition of “open source” —
with which OSI was entrusted — now differs in significant
ways from the views of most software freedom advocates.

There has been substantial acrimony during the drafting process of OSAID, and this blog post does not summarize all the
community complaints about the OSAID and its drafting
process. Other
bloggers
and the
press have covered those. The
TLDR here,
IMO is simply stated: the OSAID fails to
require reproducibility by the
public of the scientific process of building these systems, because the OSAID fails to place sufficient
requirements on the licensing and public disclosure of training sets for so-called “Open Source” systems. The
OSI refused to add this requirement because of a fundamental flaw in their process; they decided that “there
was no point in publishing a definition that no existing AI system could
currently meet”. This fundamental compromise undermined the community process, and amplified the role of stakeholders who would financially benefit from OSI’s retroactive declaration that their systems are “open source”. The OSI should have refrained from publishing a definition yet, and instead
labeled this document as ”recommendations” for now.

As the publication date of the OSAID approached, I could not help but
remember a fascinating statement that Donald E. Knuth, one of the founders
of the field of computer
science, once
said: [M]y role is to be on the bottom of things. … I try to
digest … knowledge into a form that is accessible to people who don’t
have time for such study. If we wish to engage in the
highly philosophical (and easily politically corruptible) task
of defining what terms like “software freedom” and
“open source” mean, we must learn to be on the “bottom of
things”. OSI made an unforced error in this regard. While they could
have humbly announced this as “recommendations” or “guidelines”,
they instead formalized it as a “definition” — with equivalent authority to their
OSD.

Yet, OSI itself only turned its attention to AI only recently, when they
announced their “deep dive” — for which Microsoft’s GitHub was OSI’s “Thought Leader”.
OSI has responded too rapidly to this industry ballyhoo. Their celerity of response made OSI
an easy target for regulatory capture.

By comparison, the original OSD was first published in February 1999.
That was at least twelve years after the widespread industry adoption of
various FOSS programs (such as the GNU C Compiler and BSD). The concept was explored and discussed publicly (under the moniker “Free Software”)
for decades before it was officially “defined”.
The OSI announced itself as the “marketing department for Free Software” and
based the OSD in large part on the independently
developed Debian Free Software Guidelines (DFSG). The OSD was thus the
culmination of decades of thought and consideration, and primarily developed
by a third-party (Debian) — which provided a balance on OSI’s authority.
(Interestingly, some folks from Debian are attempting to check OSI’s authority again due to the premature publication of the OSAID.)

OSI claims that they must move quickly so that they can
counter the software companies from coopting
the term “open source” for their own aims. But
OSI failed to pursue trademark protection for “open source” in the early days, so the OSI can’t stop Mark Zuckerberg and his
cronies in any event from using the “open source”
moniker for his Facebook and Instagram products — let alone his
new Llama product.
Furthermore, OSI’s insistence
that the definition was urgently needed and that the definition
be engineered as a retrofit to apply to an existing, available system has yielded troublesome results.
Simply put, OSI has a tiny sample set to examine, in 2024,
of what LLM-backed generative AI systems look like. To make a final decision
about the software freedom and rights implications of such a nascent field led to
an automatic bias to accept the actions of first movers as legitimate.
By making this definition official too soon, OSI has endorsed demonstrably bad LLM-backed generative AI systems
as “open source” by definition!

OSI also disenfranchised the users and content creators in this process.
FOSS activists should
be engaging with
the larger discussions with
impacted communities of content creators about what “open
source” means to them, and how they feel about incorporation of
their data in the training sets into these third-party systems. The line between data and code is so easily crossed with
these systems that we cannot rely on old, rote conclusions that the
“data is separate and can be proprietary (or even unavailable), and yet the system remains ‘open
source’”. That adage fails us when analyzing this technology,
and we must take careful steps — free from the for-profit corporate
interest of AI fervor — as we decide how our well-established
philosophies apply to these changes.

FOSS activists err when we unilaterally dictate and define what is
ethical, moral, open and Free in areas outside of software. Software rights
theorists can (and should) make meaningful contributions in these
other areas, but not without substantial collaboration with those creative
individuals who produce the source material. Where were the painters, the
novelists, the actors, the playwrights, the musicians, and the poets in the
OSAID drafting process? The OSD was (of course) easier because our
community is mostly programmers and developers (or folks adjacent
to those fields); software creators knew best how to consider philosophical implications of pure software products.
The OSI, and the folks in its leadership, definitely
know software well, but I wouldn’t name any of them (or myself) as great
thinkers in these many areas outside software that are noticeably impacted by the promulgation of
LLMs that are trained on those creative works. The Open Source community remains
consistently in danger of excessive insularity, and the OSAID is an
unfortunate example of how insular we can be.

Meanwhile, I have spent literally months of time over the last 30 years trying to make sure the
coalition of software freedom & rights activists remained in basic
congruence (at least publicly) with those (like OSI) who are oriented towards a more
for-profit and corporate open source approach. Until today, I was always able to say:
“I believe that anything the OSI calls ‘open source’
gives you all the rights and freedoms that you deserve”. I now cannot
say that again unless/until the OSI revokes the OSAID. Unfortunately, that
Rubicon may have now been permanently crossed! OSI
has purposely made it politically unviable for them to
revoke the OSAID. Instead, they plan only incremental updates to the OSAID. Once
entities begin to rely on this definition as written, OSI will find it nearly impossible to
later declare systems that were “open source” under 1.0 as no longer so (under later versions). So, we are likely stuck
with OSAID’s key problems forever. OSI undermines its position as a philosophical leader in Open Source as long as OSAID 1.0 stands as a formal defintion.

I truly don’t know for sure (yet) if the only way to respect user rights in an LLM-backed
generative AI system is to only use training sets that are publicly
available and licensed under Free Software licenses. I do believe
that’s the ideal and preferred form for modification of those systems. Nevertheless,
a generally useful technical system that is built by collapsing data (in essence, via highly lossy compression) into a table of floating point numbers
is philosophically much more complicated than binary software and its Corresponding Source. So, having
studied the issue myself, I believe the Socratic Epiphany currently applies. Perhaps there is an acceptable
spot for compromise
regarding the issues of training set licensing, availability and similar reproducibility issues.
My instincts, after 25
years as a software rights philosopher, lead me to believe that it will
take at least a decade for our best minds to find a reasonable answer on where the bright line is of
acceptable behavior with regard to these AI systems. While OSI claims their OSAID is humble, I beg
to differ. The humble act now is to admit that it was just too soon to publish a “definition” and
rebrand these the OSAID 1.0 as “current recommendations”. That might not grab as many
headlines or raise as much money as the OSAID did, but it’s the moral and ethical way out of this bad situation.

Finally, rather than merely be a pundit on this matter, I am instead today putting myself forward
to try to be part of the solution. I plan to run for the OSI Board of Directors at the next elections on a single-issue
platform: I will work arduously for my entire term to see the OSAID repealed, and republished
not as a definition, but merely recommendations, and to also issue a statement
that OSI published the definition sooner than was appropriate. I’ll write further about the matter as the
next OSI Board election approaches. I also call on other software rights activists to run with me on a similar platform; the OSI has myriad seats that are elected by different constituents, so there is opportunity to run as a ticket on this issue. (Please contact me privately if you’d like to be involved with this ticket at the next OSI Board election. Note, though, that election results
are not actually binding, as OSI’s by-laws allow the current Board to reject results of the elections.)

HPE Cray XD670 NVIDIA HGX with CoolIT Liquid Cooling Shown

Post Syndicated from Eric Smith original https://www.servethehome.com/hpe-cray-xd670-nvidia-hgx-coolit-liquid-cooling-gigabyte-intel-ocp-shown/

We take a look at the HPE Cray XD670, a NVIDIA HGX H100 and H200 system with CoolIT liquid-cooling and discuss how it compares

The post HPE Cray XD670 NVIDIA HGX with CoolIT Liquid Cooling Shown appeared first on ServeTheHome.

Streamline AI-driven analytics with governance: Integrating Tableau with Amazon DataZone

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/streamline-ai-driven-analytics-with-governance-integrating-tableau-with-amazon-datazone/

Amazon DataZone is a data management service that makes it faster and easier for customers to catalog, discover, share, and govern data stored across AWS, on premises, and from third-party sources. Amazon DataZone recently announced the expansion of data analysis and visualization options for your project-subscribed data within Amazon DataZone using the Amazon Athena JDBC driver.

Collaborating closely with our partners, we have tested and validated Amazon DataZone authentication via the Athena JDBC connection, providing an intuitive and secure connection experience for users. With this integration, you can now seamlessly query your governed data lake assets in Amazon DataZone using popular business intelligence (BI) and analytics tools, including partner solutions like Tableau.

Ali Tore, Senior Vice President of Advanced Analytics at Salesforce, highlighting the value of this integration, says

“We’re excited to partner with Amazon to bring Tableau’s powerful data exploration and AI-driven analytics capabilities to customers managing data across organizational boundaries with Amazon DataZone. This integration enables our customers to seamlessly explore data with AI in Tableau, build visualizations, and uncover insights hidden in their governed data, all while leveraging Amazon DataZone to catalog, discover, share, and govern data across AWS, on premises, and from third-party sources—enhancing both governance and decision-making.”

With this launch, Amazon DataZone strengthens its commitment to empowering enterprise customers with secure, governed access to data across the tools and platforms they rely on. For example, Guardant Health uses Amazon DataZone to democratize data access across its organization, enabling diverse teams to efficiently access, query, and analyze data tailored to their specific needs.

Rajesh Kucharlapati, Senior Director of Data, CRM, and Analytics at Guardant Health, says

“By harmonizing data across multiple business domains, we foster a culture of data sharing. Using Amazon DataZone lets us avoid building and maintaining an in-house platform, allowing our developers to focus on tailored solutions. Leveraging AWS’s managed service was crucial for us to access business insights faster, apply standardized data definitions, and tap into generative AI potential. We also needed an easy connection process for widely-used analytics tools like Tableau, DBeaver, and Domino, directly within Amazon DataZone projects. This new JDBC connectivity feature enables our governed data to flow seamlessly into these tools, supporting productivity across our teams.”

Use case

Amazon DataZone addresses your data sharing challenges and optimizes data availability. Here’s how:

  • Data product creation – As a data producer, you can create and catalog data products while enforcing governance, making your data findable, accessible, interoperable, and reusable (FAIR).
  • Streamlined access – As a data consumer, you can easily locate and subscribe to data from multiple sources within a single project. You can analyze this data using a variety of tools, including built-in AWS options such as Amazon Athena, Amazon Redshift, and Amazon SageMaker.
  • Integration with partner tools – The addition of support for partner analytics tools offers you greater flexibility and efficiency in your workflows. You can now use your tool of choice, including Tableau, to quickly derive business insights from your data while using standardized definitions and decentralized ownership. Refer to the detailed blog post on how you can use this to connect through various other tools.

Prerequisites

To get started, complete these steps:

  1. Download and install the latest Athena JDBC driver for Tableau.
  2. Copy the JDBC connection string from the Amazon DataZone portal into the JDBC connection configuration to establish a connection from Tableau. This will direct you to authenticate using single sign-on with your corporate credentials.

When you’re connected, you can query, visualize, and share data—governed by Amazon DataZone—within Tableau.

The following diagram shows the high-level architecture of the Tableau integration.

Solution walkthrough: Configure Tableau to access project-subscribed data assets

To configure Tableau to access project-subscribed data assets, follow these detailed steps:

  1. Download the latest Athena driver. If Tableau has the Athena driver preinstalled, it could be the older (v2) version. To confirm compatibility with Amazon DataZone, you’ll need the latest (v3) driver that includes the necessary authentication features. To download the latest JDBC driver version x, visit Athena JDBC 3.x driver.
  2. Install the driver. Copy the JDBC driver file to the appropriate folder for your operating system:
    • For macOS: ~/Library/Tableau/Drivers
    • For Windows: C:\Program Files\Tableau\Drivers
  3. On the Amazon DataZone console, select your project, as shown in the following screenshot of DataZone Console.
  4. To capture the JDBC connection parameters, follow these steps:
    1. On the project page, review the connection options under ANALYTICS TOOLS. Choose Connect with JDBC.
    2. In the JDBC parameters dialog box, select Using IDC auth and copy the JDBC URL. Optionally, you can use Using IAM auth to connect with your Amazon DataZone project as an AWS Identity and Access Management (IAM) role (from a server), provided that you are added as a project member within that project. The following screenshot shows the dialog box.
  5. To configure the Tableau desktop for connection, follow these steps:
    1. On the To a Server connection menu, select Other Databases (JDBC).
    2. Paste the copied JDBC URL into the URL field, leaving the other fields (Dialect, Username, Password) unchanged.
  6. To sign in with single sign-on, choose Sign in, as shown in the following screenshot. You’ll be redirected to authenticate with AWS IAM Identity Center. Use the credentials for your AWS single sign-on account.
  7. After you’re signed in, you’ll be prompted to authorize the DataZoneAuthPlugin. Choose Allow access to authorize access to Amazon DataZone from Tableau, as shown in the following screenshot.
  8. After the connection is established, a success message will appear, as shown in the following screenshot.

You can now view your project’s subscribed data directly within Tableau and build dashboards.

Conclusion

Amazon DataZone continues to expand its offerings, providing you with more flexibility in how you access, analyze, and visualize your subscribed data. With support for the Athena JDBC driver, you can now use a wide range of popular BI and analytics tools including Tableau, making governed data within Amazon DataZone more accessible than ever before.

In this post, you learned how the recent enhancements in Amazon DataZone facilitate a seamless connection with Tableau. By integrating Tableau with the comprehensive data governance capabilities of Amazon DataZone, we’re empowering data consumers to quickly and seamlessly explore and analyze their governed data. This integration helps organizations break down silos, foster collaboration, and make informed decisions, all while maintaining the security and control needed in today’s complex, distributed data landscape.

The feature is supported in all AWS commercial Regions where Amazon DataZone is currently available. Check out the video below and the detailed blog post to learn how to connect Amazon DataZone to external analytics tools via JDBC. Get started with our technical documentation.

Related blog posts

About the Authors

Ramesh H Singh is a Senior Product Manager Technical (External Services) at AWS in Seattle, Washington, currently with the Amazon DataZone team. He is passionate about building high-performance ML/AI and analytics products that enable enterprise customers to achieve their critical goals using cutting-edge technology. Connect with him on LinkedIn.

Adiascar Cisneros is a Tableau Senior Product Manager based in Atlanta, GA. He focuses on the integration of the Tableau Platform with AWS services to amplify the value users get from our products and accelerate their journey to valuable, actionable insights. His background includes analytics, infrastructure, network security, and migrations. Follow him on LinkedIn.

Joel Farvault is Principal Specialist SA Analytics for AWS with 25 years’ experience working on enterprise architecture, data governance and analytics, mainly in the financial services industry. Joel has led data transformation projects on fraud analytics, claims automation, and Master Data Management. He leverages his experience to advise customers on their data strategy and technology foundations.

Yogesh Dhimate is a Sr. Partner Solutions Architect at AWS, leading technology partnership with Tableau. Prior to joining AWS, Yogesh worked with leading companies including Salesforce driving their industry solution initiatives. With over 20 years of experience in product management and solutions architecture Yogesh brings unique perspective in cloud computing and artificial intelligence.

Ariana Rahgozar is a Sr. Senior Solutions Architect at AWS, leading customers design and implement technical solutions as part of their cloud journey.

Expanding data analysis and visualization options: Amazon DataZone now integrates with Tableau, Power BI, and more

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/expanding-data-analysis-and-visualization-options-amazon-datazone-now-integrates-with-tableau-power-bi-and-more/

Amazon DataZone  now launched authentication supports through the  Amazon Athena JDBC driver, allowing data users to seamlessly query their subscribed data lake assets via popular business intelligence (BI) and analytics tools like Tableau, Power BI, Excel, SQL Workbench, DBeaver, and more. This integration empowers data users to access and analyze governed data within Amazon DataZone using familiar tools, boosting both productivity and flexibility.

Customers use Amazon DataZone to streamline data access and governance by enabling data users to locate and subscribe to data from multiple sources within a single project. Amazon DataZone natively integrates with Amazon-specific options like Amazon Athena, Amazon Redshift, and Amazon SageMaker, allowing users to analyze their project governed data. With this launch of JDBC connectivity, Amazon DataZone expands its support for data users, including analysts and scientists, allowing them to work in their preferred environments—whether it’s SQL Workbench, Domino, or Amazon-native solutions—while ensuring secure, governed access within Amazon DataZone.

Collaborating closely with our partners, we have tested and validated Amazon DataZone authentication via the Athena JDBC connection, providing an intuitive and secure connection experience for users. With this integration, you can now seamlessly query your governed data lake assets in Amazon DataZone using popular business intelligence (BI) and analytics tools, including partner solutions like Tableau.

Ali Tore, Senior Vice President of Advanced Analytics at Salesforce, highlighting the value of this integration, says

“We’re excited to partner with Amazon to bring Tableau’s powerful data exploration and AI-driven analytics capabilities to customers managing data across organizational boundaries with Amazon DataZone. This integration enables our customers to seamlessly explore data with AI in Tableau, build visualizations, and uncover insights hidden in their governed data, all while leveraging Amazon DataZone to catalog, discover, share, and govern data across AWS, on premises, and from third-party sources—enhancing both governance and decision-making.”

With this launch, Amazon DataZone strengthens its commitment to empowering enterprise customers with secure, governed access to data across the tools and platforms they rely on. For example, Guardant Health uses Amazon DataZone to democratize data access across its organization, enabling diverse teams to efficiently access, query, and analyze data tailored to their specific needs.

Rajesh Kucharlapati, Senior Director of Data, CRM, and Analytics at Guardant Health, says

“By harmonizing data across multiple business domains, we foster a culture of data sharing. Using Amazon DataZone lets us avoid building and maintaining an in-house platform, allowing our developers to focus on tailored solutions. Leveraging AWS’s managed service was crucial for us to access business insights faster, apply standardized data definitions, and tap into generative AI potential. We also needed an easy connection process for widely-used analytics tools like Tableau, DBeaver, and Domino, directly within Amazon DataZone projects. This new JDBC connectivity feature enables our governed data to flow seamlessly into these tools, supporting productivity across our teams.”

Getting started

To get started, download and install the latest Athena JDBC driver for your tool of choice. After installation, copy the JDBC connection string from the Amazon DataZone portal into the JDBC connection configuration to establish a connection from your tool. This will direct you to authenticate using single sign-on (SSO) with your corporate credentials. After connecting, you can query, visualize, and share data—governed by Amazon DataZone—within the tools you already know and trust.

In this post, we’ll guide you through connecting various analytics tools to Amazon DataZone using the Athena JDBC driver, enabling seamless access to your subscribed data within your Amazon DataZone projects.

Solution overview

To demonstrate these capabilities, consider a use case where your marketing team wants to drive a campaign that’s focused on product adoption. To achieve this, you need access to sales orders, shipment details, and customer data owned by the retail team. The retail team, acting as the data producer, publishes the necessary data assets to Amazon DataZone, allowing you, as a consumer, to discover and subscribe to these assets.

After the subscription is approved, the data assets become available within your marketing team’s project environment in Amazon DataZone. You can then use your preferred tool (for example, DBeaver, as shown in the following diagram) to perform data exploration.

Prerequisites

To follow along with this post, you need to have the following prerequisites in place:

  1. AWS account – You must have an active AWS account. If you don’t have one, see How do I create and activate a new AWS account?.
  2. Amazon DataZone resources – You need a domain for Amazon DataZone, an Amazon DataZone project, and a new Amazon DataZone project environment (DefaultDataLake environment with a DataLakeProfile).
  3. Publish data assets – As the data producer from the retail team, you must ingest individual data assets into Amazon DataZone. For this use case, create a data source and import the technical metadata of four data assets—customers, order_items, orders, products, reviews, and shipments—from AWS Glue Data Catalog. Ensure the data assets are enriched with business descriptions and published to the catalog.
  4. Subscribe data assets – As a data analyst from the marketing team, you must discover and subscribe to the data assets. The data producer from the retail team will review and approve your subscription. Upon successful fulfillment, the data assets will be added to your data lake environment. For detailed subscription instructions, see the Amazon DataZone User Guide.

The following figure shows the subscribed assets added to the data lake environment in your marketing project.

In the following sections, we will walk you through the steps to configure DBeaver to consume the subscribed assets from Amazon DataZone.

Configuring DBeaver to access subscribed data assets

In this section, you configure DBeaver to access the subscribed assets from the Marketing project

To configure DBeaver:

  1. Connect with JDBC: In the Amazon DataZone portal, navigate to the Marketing project, select the Environments tab and select Connect with JDBC.
    1. Select Marketing from the list in the top navigation are.
    2. Choose Environments
    3. Select Connect with JDBC.

  1. A new screen will display the JDBC connection parameters. Make sure to capture these details for configuring the database connection in DBeaver, including the JDBC URL, Domain ID, Environment ID, Region, and IDC Issuer URL.
  2. Download and install the latest Athena driver:
    • If DBeaver has the Athena driver pre-installed, it might be the older (v2) version. To ensure compatibility with Amazon DataZone, you need the latest driver (v3), which includes the necessary authentication features.
    • Download the latest JDBC driver—version 3.x.
    • To install the latest driver:
      • Go to Database and then to Driver Manager in DBeaver.
      • Select the Athena driver and choose Edit.
      • Choose Download to fetch the latest driver version.
      • If prompted, select the appropriate version and confirm the download.
  1. In the DBeaver SQL client, create a new database connection and select the Athena driver.
  2. In the Driver Properties section, enter the parameters that you captured from Amazon DataZone:
    • CredentialsProvider: The credentials provider to authenticate requests to AWS
    • DataZoneDomainId: The ID of your Amazon DataZone domain
    • DataZoneDomainRegion: The AWS Region where your domain is hosted.
    • DataZoneEnvironmentId: The ID of your DefaultDataLake environment.
    • IdentityCenterIssuerUrl: The issuer URL used by AWS IAM Identity Center for token issuance.
    • OutputLocation: Amazon S3 path for storing query results.
    • Region: The Region where the environment is created.
    • Workgroup: Amazon Athena workgroup of the environment.

  1. Choose Test connection.
  2. You will be redirected to the IAM Identity Center sign-in portal. Sign in with your credentials. If you’re already signed in through single sign-on (SSO), this step will be skipped.
  3. After you sign in, you will be prompted to authorize the DataZoneAuthPlugin. Choose Allow access to authorize access to Amazon DataZone from DBeaver.
  4. After the connection is established, a success message will appear as shown in the screenshot
  5. You can now view and query all subscribed assets directly within DBeaver.

These steps might also apply to other analytics tools and clients that support JDBC connections. If you’re using a different tool, you might need to adapt these instructions accordingly to ensure proper configuration and access to Amazon DataZone data assets.

Integration with other applications

You can use similar steps for other BI and analytics tools that support standard database connections.

Connect to Tableau Desktop

Use the Athena JDBC driver to connect Tableau to Amazon DataZone and visualize your subscribed data.

To connect to Tableau Desktop:

  1. Make sure that you’re using the latest Athena JDBC 3.x driver.
  2. Copy the JDBC driver file and place it in the appropriate folders for your operating system
    • For Mac OS: ~/Library/Tableau/Drivers
    • For Windows: C:\Program Files\Tableau\Drivers 
  3. Open Tableau Desktop. From the To a Server connection menu, select Other Databases (JDBC) to connect to Amazon DataZone.
  4. Paste the JDBC connection string you copied from the DataZone portal into the URL Leave other fields such as Dialect, Username, and Password blank and choose Sign in.
  5. This will redirect you to authenticate with IAM Identity Center. Enter the credentials of the Identity Center user that you used to sign in to the DataZone portal. Authorize the DataZoneAuthPlugin to access Amazon DataZone from Tableau. Once the connection is established with the success message, you now view your project’s subscribed data directly within Tableau and build dashboards.

See the Amazon DataZone and Tableau blog post for step-by-step instructions.

Connect to Microsoft Power BI

Now, let’s look at connecting Amazon DataZone with Microsoft Power BI on Windows.

While Amazon Athena provides a native ODBC driver for connecting to ODBC-compatible tools like Microsoft Power BI, it currently doesn’t support Amazon DataZone authentication. Therefore, in this post, we will use an ODBC-JDBC bridge to connect Amazon DataZone with Microsoft Power BI using the Athena JDBC driver, which supports DataZone authentication.

In this post, we’re using the ZappySys driver as the ODBC-JDBC bridge. This is a third-party solution that requires a separate licensing fee, which isn’t included in the AWS solution. You can choose to use any other solution for ODBC-JDBC bridge.

To connect to Power BI:

  1. Make sure that you have administrator privileges to run the ODBC Data Source Administrator.
  2. From the Windows Start menu, run the ODBC Data Source Administrator (the 64-bit version) using run as Administrator.
  3. Create a New Data Source with the ZappySys JDBC Bridge Driver. You will be prompted to enter your connection details.
  4. Paste the JDBC URL you copied from the DataZone portal in the Connection String, along with the driver class and JDBC driver file. Make sure that you’re using the latest Athena JDBC 3.x driver.
  5. Choose Test Connection. A new dialog window will pop up after the connection is successful.
  6. After configuring the data source, launch Power BI. Create a blank report or use an existing report to integrate the new visuals. Choose Get Data and select the name of the data source you created. This will open a new browser window to authenticate your credentials. Allow access to authorize the DataZone plugin. After authorization is complete, you can build your reports in Microsoft Power BI with the subscribed data assets.

Connect to SQL Workbench

Discover how SQL Workbench can connect to Amazon DataZone for users who prefer a SQL interface to query data lake tables and views subscribed through projects in Amazon DataZone.

To connect to SQL Workbench

  1. Make sure that you’re using the latest Athena JDBC 3.x driver.
  2. Open SQL Workbench/J and choose Manage Drivers.
  3. Select the option to add a new driver. Enter a name for it, such as DatazoneAthenaJDBC, and import the driver you downloaded in the previous steps.
  4. Create a new connection and enter a name it, such as datazone-profile. In the Driver option, select the driver you configured.
  5. For the URL, enter the string jdbc:athena://region=us-east-1; (In the example, the Virginia Region is being used). Choose Extended Properties.
  6. Under Extended Properties, add the following parameters that you copied from the DataZone portal and choose OK. You can also include these parameters in the JDBC (URL) connection string.

    1. The parameters to add are:
      • Workgroup
      • DataZoneEndpointOverride
      • OutputLocation
      • DataZoneDomainId
      • IdentityCenterIssuerURL
      • CredentialsProvider
      • DatazoneEnvironmentId
      • DataZoneDomainRegain

  1. You will be prompted to sign in and authenticate. Allow access and authorization to Amazon DataZone.
  2. After successful connection, in SQL Workbench/J, under Database Explorer, select the desired database. For example, select the database that has access to the subscribed data asset orders. Select the data asset and execute the query.

Cleanup

To ensure no additional charges are incurred after testing, be sure to delete the Amazon DataZone domain. See Delete Amazon DataZone domains for instructions.

Conclusion

Amazon DataZone continues to expand its offerings, providing you with more flexibility to access, analyze, and visualize your subscribed data. With support for the Athena JDBC driver, you can now use a wide range of popular BI and analytics tools, making data accessed through Amazon DataZone more accessible than ever before. Whether you’re using Tableau, Power BI, or other familiar tools, the integration with Amazon DataZone ensures that your data remains secure and accessible to authorized users.

The feature is supported in all AWS commercial Regions where Amazon DataZone is currently available. Watch the video below to learn how to connect Amazon DataZone to external analytics tools via JDBC. Get started with our technical documentation.

About the Authors

Ramesh H Singh is a Senior Product Manager Technical (External Services) at AWS in Seattle, Washington, currently with the Amazon DataZone team. He is passionate about building high-performance ML/AI and analytics products that enable enterprise customers to achieve their critical goals using cutting-edge technology. Connect with him on LinkedIn.

Eric Fleishman is a software engineer at AWS in Seattle. He loves diving into cloud technology and solving complex problems to build impactful solutions. Outside of work, he is all about staying active—whether its snowboarding down the slopes or working out. He enjoys pushing his limits and embracing new challenges.

Theo Tolv is a Senior Analytics Architect based in Stockholm, Sweden. He’s worked with small and big data for most of his career, and has built applications running on AWS since 2008. In his spare time he likes to tinker with electronics and read space opera.

Joel Farvault is Principal Specialist SA Analytics for AWS with 25 years’ experience working on enterprise architecture, data governance and analytics, mainly in the financial services industry. Joel has led data transformation projects on fraud analytics, claims automation, and Master Data Management. He leverages his experience to advise customers on their data strategy and technology foundations.

Lakshmi Nair is a Senior Analytics Specialist Solutions Architect at AWS. She specializes in designing advanced analytics systems across industries. She focuses on crafting cloud-based data platforms, enabling real-time streaming, big data processing, and robust data governance.

Fabricio Hamada is a Senior Data Strategy Solutions Architect at AWS.

Lionel Pulickal is Sr. Solutions Architect at AWS

The collective thoughts of the interwebz

Proudly powered by Ants