Tracking Down a Suspect through Cell Phone Records

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/07/tracking-down-a-suspect-through-cell-phone-records.html

Interesting forensics in connection with a serial killer arrest:

Investigators went through phone records collected from both midtown Manhattan and the Massapequa Park area of Long Island—two areas connected to a “burner phone” they had tied to the killings. (In court, prosecutors later said the burner phone was identified via an email account used to “solicit and arrange for sexual activity.” The victims had all been Craigslist escorts, according to officials.)

They then narrowed records collected by cell towers to thousands, then to hundreds, and finally down to a handful of people who could match a suspect in the killings.

From there, authorities focused on people who lived in the area of the cell tower and also matched a physical description given by a witness who had seen the suspected killer.

In that narrowed pool, they searched for a connection to a green pickup truck that a witness had seen the suspect driving, the sources said.

Investigators eventually landed on Heuermann, who they say matched a witness’ physical description, lived close to the Long Island cell site and worked near the New York City cell sites that captured the other calls.

They also learned he had often driven a green pickup truck, registered to his brother, officials said. But they needed more than just circumstantial evidence.

Investigators were able to obtain DNA from an immediate family member and send it to a specialized lab, sources said. According to the lab report, Heuermann’s family member was shown to be related to a person who left DNA on a burlap sack containing one of the buried victims.

There’s nothing groundbreaking here; it’s casting a wide net with cell phone geolocation data and then winnowing it down using other evidence and investigative techniques. And right now, those are expensive and time consuming, so only used in major crimes like murder (or, in this case, murders).

What’s interesting to think about is what happens when this kind of thing becomes cheap and easy: when it can all be done through easily accessible databases, or even when an AI can do the sorting and make the inferences automatically. Cheaper digital forensics means more digital forensics, and we’ll start seeing this kind of thing for even routine crimes. That’s going to change things.

Wiwynn Liquid Cooling for 8kW AI Accelerator Trays Shown

Post Syndicated from Eric Smith original https://www.servethehome.com/wiwynn-liquid-cooling-for-8kw-of-ai-accelerators-shown-oam-oai-ocp-ubb/

We check out the Hyper-scale server maker Wiwynn’s liquid cooling solution for 8kW GPU and AI accelerator trays at its headquarters

The post Wiwynn Liquid Cooling for 8kW AI Accelerator Trays Shown appeared first on ServeTheHome.

Gigabyte R263-P33 Ampere AmpereOne Arm Server at Computex 2023

Post Syndicated from Cliff Robinson original https://www.servethehome.com/gigabyte-r263-p33-ampere-ampereone-arm-server-at-computex-2023/

We spotted the new Gigabtye R263-P33 Ampere AmpereOne Arm server and wanted to show our readers the upcoming Arm server platform

The post Gigabyte R263-P33 Ampere AmpereOne Arm Server at Computex 2023 appeared first on ServeTheHome.

Защо е толкова важен законът за несъстоятелност на физичските лица

Post Syndicated from VassilKendov original http://kendov.com/%D0%B7%D0%B0%D1%89%D0%BE-%D0%B5-%D1%82%D0%BE%D0%BB%D0%BA%D0%BE%D0%B2%D0%B0-%D0%B2%D0%B0%D0%B6%D0%B5%D0%BD-%D0%B7%D0%B0%D0%BA%D0%BE%D0%BD%D1%8A%D1%82-%D0%B7%D0%B0-%D0%BD%D0%B5%D1%81%D1%8A%D1%81%D1%82/

Защо е толкова важен законът за несъстоятелност на физичските лица

Една от основните цели на такъв закон трябва да бъде уреждането на отношенията между кредитор и кредитоискател, в случаите когато кредита не може да бъде обслужван. Но това не е достатъчно. И сега имаме процедура за уреждане на тези взаимоотношения. В общия случай това е съд и после ЧСИ.
Проблемът идва от това, че при сегашната процедура единствения начин кредитоискателя да продължи да води достоен живот е, да емигрира или да излезе в сивата икономика. И в двата случая обществото губи.
Какво губи ли? Губим данъците които той би трябвало да внася, ако не работи в сивата икономика. Губим неговите разходи за живот и потребление ако той е емигрирал. Губим потребелението на неговото семейство (съпруа/съпруг, деца), губим население ако замине в чужбина.
А какво, колко и кой печели от нередовния кредит?

Със сигурност печели ЧСИ. Те получават парите си предварително от възложителя (банките). За банката и другите кредитори не е сигурно колко и дали ще спечелят. Банките например начисляват лихва за просрочие, но дали ще я съберат? Да, дадена им е възможност да си ги събират до живот, но това е още една причина за кредитоискателя да напусне България или да излезе в сивата икономика до живот.
Верно е, че се въведе 10 годишна абсолютна давност, но ако се замислите 10 години са достатъчно дълъг период за да си заслужава да вземеш мерки и да потърсиш варианти за живот в чужбина или в сивата икономика.

С две думи, за да осигурим по-дълъг период на кредиторите за събиране на вземанията, ние рискуваме намаляне на населението и увеличение на сивата икономика, комбинирано с по-ниски приходи в бюджета. Тояно това трябва да се регулира от един такъв закон за Несъстоятелност на физическите лица.

В момента сме на етап обсъждане на предложения закон от Крум Зарков, между първо и второ четене в парламента. В предходни статии влязох в детайли относно неприложимостта на този закон, но след като се запознах с предложенията за корекции на ГЕРБ, ППДБ и Възраждане, съвсем се отчаях.

На практика ние приемаме закон, по който никой или поне много малка част от хората биха обявили фалит, просто защото не попадат в хипотезите на този закон или защото просто няма да им е изгодно да подемат таква процедура. Този закон просто няма да работи и това смятам е заложено съвем умишлено. Единственият мотив за гласуването му е изискването да го приемем, за да ни отпуснат следващия транш по програмата за възстановяване и устоичивост в размер на 724 млн EUR. Сега просто се взимат мерки, той да не промени дефакто нищо в системата на събиране на дълговете.

За  срещa с мен моля използвайте посочената форма.

[contact-form-7]

Преди да обърна внимание на предложенията за промени на отделните политически партии, искам да отбележа, че имаме отрицателни становища за този закон от всички възможни асоциации и институции – Камара на ЧСИ, БНБ, ВКС, СГС, Адвокатска колегия…
Странното е, че в момента при обсъжданията на закона никой не взема предвид тези коментари, а се правят някакви абсурдни предложения. Ето някои примери

ППДБ

„Предложението на депутатите от ППДБ е човек да влиза в затвора за срок до три години за умишлен банкрут, ако поиска от съда обявяване в несъстоятелност и след това скрие или даде на близки или роднини пари или ценни вещи, които може да бъдат взети от кредиторите”

„да не бъдат опрощавани задължения на човек, който 5 години преди да поиска обявяване в несъстоятелност е продал вещ или имущество на занижена цена. Идеята на промените е да бъдат пресечени опити на длъжници умишлено да увредят интересите на кредиторите, гласят мотивите.”

Аз не мога да си представя кой би планувал личен фалит цели 5 години, но това е защото се занимавам с лоши кредити и консултации на фирми по занятие. Депутатите от тази партия явно имат друг житейски опит

ГЕРБ СДС

„законът за фалит на граждани да бъде прилаган само при дългове в размер над 30 минимални заплати – 23 400 лева към момента.”
В закона е предвидено дългът да е над 10 минимални заплати, тоест 7 800 лева.
трябва да признаем, че това не е неразумно предложение, но далеч не е най-големият проблем в този закон.

„Да не бъдат опрощавани и дълговете на хора, които са продали свое имущество на значителна стойност на свързано лице през последните две години преди подаване на молба за откриване на производство по несъстоятелност”
Според мен най-естественото решение когато имаш финансови проблеми е да продадеш нещо за да ги решиш. Направите ли го обаче, няма да можете да обявите фалит. И кой точно печели от това? А как точно печели?

Възраждане

„Законът да бъде прилагат да задължения в размер над 5 минимални заплати, тест за борчове над 3900 лв.”
„целият процес по обявяване на човек в несъстоятелност изцяло да бъде движен от съда, а не от синдик”

Не знам как да коментирам тази идея, но да занимаваш съда с несъстоятелност за суми под 3900 лева ми се вижда не малко, а доста нерационално. Ако знаете каква е процедурата по обявяването във фалит предвидена в закона, няма да е трудно да предвидите, че съда ще бъде задръстен само с подобни дела.
Аз например отново бих попитал защо изобщо е нужен синдик, на който се плаща допълнително, за да се ликвидира имущество от един апартамент, кола и някакви идеални части от наследствени ниви, което е норамалното имущество на 99% от гражданите в България. Ами то системата с ЧСИ си работи перфектно в тези случаи. Защо трябва да се начисляват върху дълга и задълженията към синдика?

Но за синдика никой нито дума в дебатите, нищо че дублира дейноста на ЧСИ и съответно хонорарите.

За съжаление този законопроект е толкова зле напраен, че не става за дебат или промяна. Той просто няма да функционира и да балансира икономиката. Напротив, той отнема възможността да се обявяват фалити на физическите лица, като същевременно няма да може да бъде изменян през годините, за да бъде донапасван. Естествено приемането му ще доведе до получаването на следващ транш по програмата за възстановяване и устойчивост, който дори е заложен в бюджета.

Съвестът ми в тази насока е един единствен – не разчитайте на този закон и вземете навременни  мерки ако усетите невъзможност да изплащате задълженията си.

Васил Кендов – финансов съветник

Вечният длъжник остава, въпреки новия закон за несъстоятелност на физическите лица. Кой може да се възползва от него на практика?

The post Защо е толкова важен законът за несъстоятелност на физичските лица appeared first on Kendov.com.

Седмицата (10–15 юли)

Post Syndicated from Йовко Ламбрев original https://www.toest.bg/sedmitsata-10-15-yuli/

Седмицата (10–15 юли)

След емоциите и реакциите, свързани с посещението на президента на Украйна у нас, тази седмица направо изглежда скучна и в някаква степен твърде прагматична. Дотолкова, че почти четири месеца след началото на 49-тото Народно събрание и съвсем малко преди лятната ваканция постоянните парламентарни комисии най-накрая се сдобиха с председатели, а изглежда, че водещите партии са близо до разбирателство и за назначенията в т.нар. втори ешелон – заместник-министри, областни управители и ръководители на регулатори и държавни агенции. Емилия Милчева обобщава най-важното в този контекст с крайно необходимия критичен поглед в коментара си „Вече няма нито „добри“, нито „лоши“.

Правителството също вече навърши първия си месец, откакто пое кормилото на държавата, а през седмицата станаха известни резултатите от няколко социологически изследвания на обществените нагласи и оценки за него. Във връзка с това ви предлагаме анализа на Светла Енчева, озаглавен „Новото правителство: Падащи рейтинги и важни решения“.

Междувременно Европейският парламент, и то с огромно мнозинство, призова за пореден път България и Румъния да бъдат приети в Шенгенското пространство. Настоява се това да стане преди края на 2023 г., с аргументи за пропуснати финансови ползи, и се припомня, че сегашното положение нарушава Хартата на основните права на ЕС. Проблем пред уж пооткрехнатата врата към Шенген обаче може да се окаже отново Нидерландия, чието правителство подаде оставка преди броени дни.

Иначе в понеделник Румен Радев изплака, че срещата с президента Зеленски му е дошла изневиделица и не бил подготвен за нея. Използва повода да атакува помощта за Украйна и евентуалната сделка за реакторите на АЕЦ „Белене“ и разбира се, да налее масло в огъня от колебания относно готовността на България за еврозоната. Едновременно неприятно, но и полезно е, че става все по-трудно да се открият съществени разлики между това, което имаме за президент, и един друг гръмогласен политик, свързван с дребни руски монети и други дребни… неща. Но е повече от очевидно, че занапред Радев възнамерява да търси и използва всеки момент на медийно внимание за преповтаряне на познатите си тези за атака срещу кабинета „Денков“. И все повече да преминава граници, ако съдим по изявите му от Роженския събор през последните два дни. Стратегията на аргументирана контраатака, която премиерът Денков приложи, засега изглежда най-добрата възможна отбрана.

Но стига толкова вътрешнополитически теми. И бездруго жегата е достатъчно тегава. Независимо дали сте убедени, че човешките действия и бездействия са важен фактор за глобалното затопляне, планетата ни става все по-гореща, а тази година може да се окаже рекордна (поне откакто се водят изследвания). Михаил Ангелов обяснява причините за това в последните си „Научни новини“ и ни разказва за новите геномни техники за въвеждане на нови сортове, които биха били ключови за изхранването на човечеството на фона на настъпващите климатични промени.

Георги Велев пък неотдавна се завърна от пътуване в Андалусия и вглеждайки се в добрите градоустройствени практики, приложени там, неминуемо прави паралели с любимия си роден Пловдив. Макар вторият град в България да потъна в тежък застой по време на сегашното управление на кмета Здравко Димитров, според Георги са необходими няколко малки стъпки и немного средства, за да стане Пловдив повече „град за хората“. Както и разбира се, единомислие с бъдещите управници, които ще бъдат избрани наесен.

Таня Иванова е нова авторка в „Тоест“, но е сред знаковите гласове на радио Jazz FM, където води предаването „Следобедни импровизации“. Тя дебютира при нас с един вълнуващ текст, в който търси отговора на много важен въпрос: как да живеем, когато изведнъж хоризонтът ни рязко се скъси? Историята на Сулейка Джауад показва, че парадоксално, и най-тежката болест може да заздрави разкъсаната социална тъкан и да се превърне в ценност, не по-малка от здравето. В урок по живот.

И накрая, ако не сте успели да стигнете до 111-метровия пилон на връх Рожен, един клик ви дели от 112-метровия цифров пилон. Той е създаден с основна цел „да покаже, че българската дигитална мисъл няма граница, защото пилонът без проблем може да расте и да се смалява“, съобщават от PZdnes.

Приятно четене!

Metasploit Weekly Wrap-Up

Post Syndicated from Navya Harika Karaka original https://blog.rapid7.com/2023/07/14/metasploit-weekly-wrap-up-19/

Authentication bypass in WordPress Plugin WooCommerce Payments

Metasploit Weekly Wrap-Up

This week’s Metasploit release includes a module for CVE-2023-28121 by h00die. This module can be used against any wordpress instance that uses WooCommerce payments < 5.6.1. This module exploits an auth by-pass vulnerability in the WooCommerce WordPress plugin. You can simply add a header to execute the bypass and use the API to create a new admin user in WordPress.

New module content (3)

WordPress Plugin WooCommerce Payments Unauthenticated Admin Creation

Authors: Julien Ahrens, Michael Mazzolini, and h00die
Type: Auxiliary
Pull request: #18164 contributed by h00die
AttackerKB reference: CVE-2023-28121

Description: This module exploits an auth by-pass vulnerability in the WooCommerce WordPress plugin. By sending a speciality crafted request to the plugin an attacker can by-pass authentication and then use the WordPress API to create an admin user in WordPress.

pfSense Restore RRD Data Command Injection

Author: Emir Polat
Type: Exploit
Pull request: #17861 contributed by emirpolatt
AttackerKB reference: CVE-2023-27253

Description: This module exploits a vulnerability in pfSense version 2.6.0 and below which allows for authenticated users to execute arbitrary operating systems commands as root.

SmarterTools SmarterMail less than build 6985 – .NET Deserialization Remote Code Execution

Authors: 1F98D, Ismail E. Dawoodjee, and Soroush Dalili
Type: Exploit
Pull request: #18170 contributed by ismaildawoodjee
AttackerKB reference: CVE-2019-7214

Description: Adds a new module for SmarterMail Build 6985 – dotNET Deserialization Remote Code Execution (CVE-2019-7214). The vulnerability affects SmarterTools SmarterMail Version less than or equal to 16.3.6989.16341 (all legacy versions without a build number), or SmarterTools SmarterMail Build less than 6985.

Enhancements and features (0)

None

Bugs fixed (0)

None

Documentation added (2)

  • #18177 from ismaildawoodjee – Updates the Wiki to use https://metasploit.com/download instead of http://metasploit.com/download.
  • #18181 from hahwul – Updates broken links in the Wiki.

You can always find more documentation on our docsite at docs.metasploit.com.

Get it

As always, you can update to the latest Metasploit Framework with msfupdate
and you can get more details on the changes since the last blog post from
GitHub:

If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest.
To install fresh without using git, you can use the open-source-only Nightly Installers or the
binary installers (which also include the commercial edition).

Build and share a business capability model with Amazon QuickSight

Post Syndicated from Abdul Qadir original https://aws.amazon.com/blogs/big-data/build-and-share-a-business-capability-model-with-amazon-quicksight/

The technology landscape has been evolving rapidly, with waves of change impacting IT from every angle. It is causing a ripple effect across IT organizations and shifting the way IT delivers applications and services.

The change factors impacting IT organizations include:

  • The shift from a traditional application model to a services-based application model (SaaS, PaaS)
  • The shift from a traditional infrastructure and hardware costing model to cloud-based containers (private and public clouds) with metered usage for resources (IaaS)
  • The shift from the lengthy traditional development and delivery cycles to continuous development and integration (DevOps)
  • The shift in application architecture from N-Tier to loosely coupled services

The portfolio of services delivering business capabilities are the new assets of IT organizations that need to be cataloged in a repository. The system must follow a well-defined business taxonomy that enhances discovery, analysis, and reuse by potential consumers, and avoids building redundant services. The traditional portfolio management tools within the organization need to be augmented with additional components that can manage the complexity of the services ecosystem.

This post provides a simple and quick way of building an extendable analytical system using Amazon QuickSight to better manage lines of business (LOBs) with a detailed list of business capabilities and APIs, deep analytical insights, and desired graphical visualizations from different dimensions. In addition, this tool enhances the discovery and reuse of existing business capabilities, avoids duplication of services, and shortens time-to-market.

Use case overview

Bob is a Senior Enterprise Architect. He recently joined a Tier 1 bank. His first assignment is to assess the bank’s capabilities to offer new financial products to its high-value retail clients. The only document given to Bob was PowerPoint slides and the names of the head of each department to get more information. The PowerPoint presentation provided high-level information, but it didn’t give an insight into how capable each department is to provide the required data through APIs for the new products. To collect that information, Bob gets in touch with the head of each department, who in turn refer him to their development leads, who in turn give him a bunch of technical documents that explain how APIs are being used.

Relevance

Business analysts are familiar with business terminology and taxonomy, and often depend on the technology team to explain the technical assets associated with business capabilities. The business capabilities are the assets of the IT organization that need to be cataloged in a repository. The catalog must follow a well-defined business taxonomy that enhances discovery and reuse by consumers, and avoids building redundant services.

The better organized the catalog is, the higher the potential for reuse and the return on investment for the services transformation strategy. The catalog needs to be organized using some business functions taxonomy with a detailed list of capabilities and sub-capabilities. The following diagram illustrates an example of services information and interdependencies.

Example of services information and interdependencies

Defining and capturing a business capability model

If an enterprise doesn’t have a system to capture the business capability model, consider defining and finding a way to capture the model for better insight and visibility, and then map it with digital assets like APIs. The model should be able to showcase to LOBs their categories and capabilities. The following table includes some sample LOBs and their associations for a business that sells the services.

LOB

Category

Capability

Recruitment

Manage Applicant Experience

Manage Application Activities

Process Application

Follow-Ups

Pursue Automated Leads

Sale Service

Engage Customer

Provide Needs Assessment Tools

Provide Service Information

After the map is defined and captured, each business capability can be mapped to APIs that are implemented for it. Each business capability then has visibility into all the associated digital assets and mapped metadata of the services, such as consumers of the API.

To capture the model, you can define a simple table to capture the information, and then you can perform further analysis on it with an analytical tool such as QuickSight.

In the following sample data model, each business LOB has several business categories and capabilities, and each capability can be mapped to multiple APIs. Also note that there’s not always a 1:1 mapping between a business capability, an API, and a service.

  • Business LOB – Recruitment, Sale Service
  • Business category – Process Application, Engage Customer
  • Business capabilities – Complete an Application, Follow-Ups
  • Digital assets – Recruitment API, Sale Service API

There are sets of other standard information that you can include in a data model, such as API consumers.

The following example shows a table structure to capture this information.

LOB table structure

The following figure visualizes the business capabilities and associated APIs.

Visualization of business capabilities and associated APIs

The remainder of the post highlights the key components to build the full solution end to end. The UI captures the business capabilities and associated APIs, and publishes the service information through a DevOps process. The solution also includes storage and a reporting tool that complement the applications portfolio management capability in place and expand its capabilities with the services portfolio.

Aligning APIs to a business capability model

To align APIs to a business capability model, you can follow these steps:

  1. Understand the business capabilities – Identify the key business capabilities of your organization and understand how they support the overall business strategy.
  2. Map the APIs to the capabilities – Review the existing APIs and map them to the corresponding business capabilities. This will help identify any gaps in the capabilities that can be addressed through new or updated APIs.
  3. Prioritize the APIs – Prioritize the development of new or updated APIs based on their importance to the business capabilities. This will ensure that the most critical capabilities are supported by the APIs.
  4. Implement governance – Implement a governance process to ensure that the APIs are aligned with the business capabilities and are used correctly. This can include setting standards for how the APIs are designed, developed, and deployed.
  5. Monitor and measure – Monitor the usage and performance of the APIs to measure their impact on the business capabilities. Use this information to make decisions about changes to the APIs over time.
  6. Regularly review and update – Review and update the mapping of the APIs to the business capabilities on a regular basis to ensure they remain aligned with the organization’s goals and objectives.

Maintenance and evolution of a business capability model

Building a business capability model is not a one-time exercise. It keeps evolving with business requirements and usage. Data management best practices should be followed as per your company’s guidelines to have consistent data end to end.

Solution overview

In this section, we introduce the ability to capture the business capabilities and associated APIs and make them available using the QuickSight business intelligence (BI) tool, and highlight its features.

The following approach provides the ability to manage business capability models and enable them to link business capabilities with enterprise digital assets, including services, APIs, and IT systems. This solution enables IT and business teams to further drill down into the model to see what has been implemented. These details provide value to architects and analysts to assess which services can be combined to provide new offerings and shorten time-to-market, enable reusability by consumers, and avoid building redundant services.

The following key components are required:

Organizations can use their existing UI framework (if available) to capture the information, or they can use one of the open-source services available in the market. Depending on the selection and capability of the open-source product, a user interface can be generated and customized.

Let’s look at each service in our solution in more detail:

  • Amplify – Amplify is a set of tools and services that can be used together or on their own, to help front-end web and mobile developers build scalable full stack applications, powered by AWS. With Amplify, you can configure app backends and connect your app in minutes, deploy static web apps in a few clicks, and easily manage app content outside the AWS Management Console. Amplify supports popular web frameworks including JavaScript, React, Angular, Vue, and Next.js, and mobile platforms including Android, iOS, React Native, Ionic, and Flutter. Get to market faster with AWS Amplify.
  • AppSync – AWS AppSync simplifies application development by creating a universal API for securely accessing, modifying, and combining data from multiple sources. AWS AppSync is a managed service that uses GraphQL so that applications can easily get only the data they need.
  • Athena – Athena is an interactive query service that makes it easy to analyze data directly in Amazon Simple Storage Service (Amazon S3) using standard SQL. In this solution, we use Athena as a data source for QuickSight.
  • Amazon Cognito – Amazon Cognito delivers frictionless customer identity and access management (CIAM) with a cost-effective and customizable platform. It easily connects the web application to the backend resources and web services.
  • DynamoDB – DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale. DynamoDB offers built-in security, continuous backups, automated multi-Region replication, in-memory caching, and data import and export tools.
  • QuickSight – QuickSight is a serverless, cloud-based BI and reporting service that brings data insights to your teams and end-users through machine learning (ML)-powered dashboards and data visualizations, which can be accessed via QuickSight or embedded in apps and portals that your users access.

The following diagram illustrates the solution architecture.

Business capabilities insights solution architecture

In the following sections, we walk through the implementation and end-to-end integration steps.

Build a serverless web application with Amplify

The open-source Amplify provides a CLI, libraries, UI components and Amplify hosting to build full stack iOS, Android, Flutter, Web, and React Native apps. For instructions on building a serverless web application, refer to the following tutorial. For this post, we created the following GraphQL schema with amplify add api:

type BusinessCapability @model {
  company_id: ID!
  company_name: String!
  company_desc: String!
  lob_name: String!
  categoray: String!
  capability: String!
  digital_asset_type: String!
  digital_asset_name: String!
  digital_asset_info: String!
}

After we use Amplify to deploy the API in the cloud, a corresponding AppSync API and a DynamoDB table are created automatically.

You can use the Amplify UI library to generate a business capability intake form and bind the fields to your front-end code.

Amplify studio generated form

You can add authentication to your application using Amazon Cognito by running amplify add auth.

With that, you are now hosting a serverless web application for your business capabilities securely and at scale.

Set up Athena and the Athena DynamoDB data connector

The DynamoDB table generated by Amplify stores all the business capabilities. You can set up Athena and the Athena DynamoDB data connector so that you can query your tables with SQL. For more information, refer to Amazon Athena DynamoDB connector.

Enable QuickSight

Enable QuickSight in your AWS account and create the datasets. The source dataset is the Athena database and table that you created earlier. To connect, you need to allow access to query Athena and Amazon S3 via the admin user interface in QuickSight. Refer to accessing AWS resources for access requirements.

Sample reports

When all the components are up and running, you can design analyses and generate reports. For more information about gathering insights from the captured data, refer to Tutorial: Create an Amazon QuickSight analysis. You can export reports in PDF, and share analyses and reports with other users. The following screenshots are reports that reflects the relationship among LOBs, business capabilities, and APIs.

The first screenshot visualizes the capabilities and associated APIs. This enables the user to identify a set of APIs, and use the same API in new similar business functions.

Business Capability Visualization 1

The following screenshot visualizes LOBs, category, and capabilities. This enables the user to easily gain insights on these relationships.

Business Capabilities Visualization 2

Best practices

The following are some best practices for business capability modeling:

  • Define clear and measurable capabilities – Each capability should be defined in a way that is clear and measurable, so that it can be tracked and improved over time.
  • Involve key stakeholders – Involve key stakeholders in the modeling process to ensure that the capabilities accurately reflect the needs of the organization.
  • Use a consistent framework – Use a consistent framework to ensure that capabilities are defined and organized in a way that makes sense for the organization.
  • Regularly review and update – Review and update the capabilities regularly to ensure they remain relevant and aligned with the organization’s goals and objectives.
  • Use visual representations – Use visual representations, like diagrams or models, to help stakeholders understand and communicate the capabilities.
  • Implement a governance process – Implement a governance process to ensure that the capabilities are being used correctly and to make decisions about changes to the capabilities over time.

Conclusion

In this post, you learned how to build a system to manage a business capability model, and discover and visualize the results in QuickSight.

We hope that companies can use this solution to manage their enterprise capability model and enable users to explore business functions available for them to use within the organization. Business users and technical architects can now easily discover business capabilities and APIs, helping accelerate the creation and orchestration of new features. With the QuickSight web interface, you can filter through thousands of business capabilities, analyze the data for your business needs, and understand the technical requirements and how to combine existing technical capabilities into a new business capability.

Furthermore, you can use your data source to gain further insights from your data by setting up ML Insights in QuickSight and create graphical representations of your data using QuickSight visuals.

To learn more about how you can create, schedule, and share reports and data exports, see Amazon QuickSight Paginated Reports.


About the authors

Abdul Qadir is an AWS Solutions Architect based in New Jersey. He works with independent software vendors in the Northeast and provides customer guidance to build well-architected solutions on the AWS cloud platform.

Sharon Li is a solutions architect at AWS, based in the Boston, MA area. She works with enterprise customers, helping them solve difficult problems and build on AWS. Outside of work, she likes to spend time with her family and explore local restaurants.

How Amazon Finance Automation built a data mesh to support distributed data ownership and centralize governance

Post Syndicated from Nitin Arora original https://aws.amazon.com/blogs/big-data/how-amazon-finance-automation-built-a-data-mesh-to-support-distributed-data-ownership-and-centralize-governance/

Amazon Finance Automation (FinAuto) is the tech organization of Amazon Finance Operations (FinOps). Its mission is to enable FinOps to support the growth and expansion of Amazon businesses. It works as a force multiplier through automation and self-service, while providing accurate and on-time payments and collections. FinAuto has a unique position to look across FinOps and provide solutions that help satisfy multiple use cases with accurate, consistent, and governed delivery of data and related services.

In this post, we discuss how the Amazon Finance Automation team used AWS Lake Formation and the AWS Glue Data Catalog to build a data mesh architecture that simplified data governance at scale and provided seamless data access for analytics, AI, and machine learning (ML) use cases.

Challenges

Amazon businesses have grown over the years. In the early days, financial transactions could be stored and processed on a single relational database. In today’s business world, however, even a subset of the financial space dedicated to entities such as Accounts Payable (AP) and Accounts Receivable (AR) requires separate systems handling terabytes of data per day. Within FinOps, we can curate more than 300 datasets and consume many more raw datasets from dozens of systems. These datasets can then be used to power front end systems, ML pipelines, and data engineering teams.

This exponential growth necessitated a data landscape that was geared towards keeping FinOps operating. However, as we added more transactional systems, data started to grow in operational data stores. Data copies were common, with duplicate pipelines creating redundant and often out-of-sync domain datasets. Multiple curated data assets were available with similar attributes. To resolve these challenges, FinAuto decided to build a data services layer based on a data mesh architecture. FinAuto wanted to verify that the data domain owners would retain ownership of their datasets while users got access to the data by using a data mesh architecture.

Solution overview

Being customer focused, we started by understanding our data producers’ and consumers’ needs and priorities. Consumers prioritized data discoverability, fast data access, low latency, and high accuracy of data. Producers prioritized ownership, governance, access management, and reuse of their datasets. These inputs reinforced the need of a unified data strategy across the FinOps teams. We decided to build a scalable data management product that is based on the best practices of modern data architecture. Our source system and domain teams were mapped as data producers, and they would have ownership of the datasets. FinAuto provided the data services’ tools and controls necessary to enable data owners to apply data classification, access permissions, and usage policies. It was necessary for domain owners to continue this responsibility because they had visibility to the business rules or classifications and applied that to the dataset. This enabled producers to publish data products that were curated and authoritative assets for their domain. For example, the AR team created and governed their cash application dataset in their AWS account AWS Glue Data Catalog.

With many such partners building their data products, we needed a way to centralize data discovery, access management, and vending of these data products. So we built a global data catalog in a central governance account based on the AWS Glue Data Catalog. The FinAuto team built AWS Cloud Development Kit (AWS CDK), AWS CloudFormation, and API tools to maintain a metadata store that ingests from domain owner catalogs into the global catalog. This global catalog captures new or updated partitions from the data producer AWS Glue Data Catalogs. The global catalog is also periodically fully refreshed to resolve issues during metadata sync processes to maintain resiliency. With this structure in place, we then needed to add governance and access management. We selected AWS Lake Formation in our central governance account to help secure the data catalog, and added secure vending mechanisms around it. We also built a front-end discovery and access control application where consumers can browse datasets and request access. When a consumer requests access, the application validates the request and routes them to a respective producer via internal tickets for approval. Only after the data producer approves the request are permissions provisioned in the central governance account through Lake Formation.

Solution tenets

A data mesh architecture has its own advantages and challenges. By democratizing the data product creation, we removed dependencies on a central team. We made reuse of data possible with data discoverability and minimized data duplicates. This also helped remove data movement pipelines, thereby reducing data transfer and maintenance costs.

We realized, however, that our implementation could potentially impact day-to-day tasks and inhibit adoption. For example, data producers need to onboard their dataset to the global catalog, and complete their permissions management before they can share that with consumers. To overcome this obstacle, we prioritized self-service tools and automation with a reliable and simple-to-use interface. We made interaction, including producer-consumer onboarding, data access request, approvals, and governance, quicker through the self-service tools in our application.

Solution architecture

Within Amazon, we isolate different teams and business processes with separate AWS accounts. From a security perspective, the account boundary is one of the strongest security boundaries in AWS. Because of this, the global catalog resides in its own locked-down AWS account.

The following diagram shows AWS account boundaries for producers, consumers, and the central catalog. It also describes the steps involved for data producers to register their datasets as well as how data consumers get access. Most of these steps are automated through convenience scripts with both AWS CDK and CloudFormation templates for our producers and consumer to use.

Solution Architecture Diagram

The workflow contains the following steps:

  1. Data is saved by the producer in their own Amazon Simple Storage Service (Amazon S3) buckets.
  2. Data source locations hosted by the producer are created within the producer’s AWS Glue Data Catalog.
  3. Data source locations are registered with Lake Formation.
  4. An onboarding AWS CDK script creates a role for the central catalog to use to read metadata and generate the tables in the global catalog.
  5. The metadata sync is set up to continuously sync data schema and partition updates to the central data catalog.
  6. When a consumer requests table access from the central data catalog, the producer grants Lake Formation permissions to the consumer account AWS Identity and Access Management (IAM) role and tables are visible in the consumer account.
  7. The consumer account accepts the AWS Resource Access Manager (AWS RAM) share and creates resource links in Lake Formation.
  8. The consumer data lake admin provides grants to IAM users and roles mapping to data consumers within the account.

The global catalog

The basic building block of our business-focused solutions are data products. A data product is a single domain attribute that a business understands as accurate, current, and available. This could be a dataset (a table) representing a business attribute like a global AR invoice, invoice aging, aggregated invoices by a line of business, or a current ledger balance. These attributes are calculated by the domain team and are available for consumers who need that attribute, without duplicating pipelines to recreate it. Data products, along with raw datasets, reside within their data owner’s AWS account. Data producers register their data catalog’s metadata to the central catalog. We have services to review source catalogs to identify and recommend classification of sensitive data columns such as name, email address, customer ID, and bank account numbers. Producers can review and accept those recommendations, which results in corresponding tags applied to the columns.

Producer experience

Producers onboard their accounts when they want to publish a data product. Our job is to sync the metadata between the AWS Glue Data Catalog in the producer account with the central catalog account, and register the Amazon S3 data location with Lake Formation. Producers and data owners can use Lake Formation for fine-grained access controls on the table. It is also now searchable and discoverable via the central catalog application.

Consumer experience

When a data consumer discovers the data product that they’re interested in, they submit a data access request from the application UI. Internally, we route the request to the data owner for the disposition of the request (approval or rejection). We then create an internal ticket to track the request for auditing and traceability. If the data owner approves the request, we run automation to create an AWS RAM resource share to share with the consumer account covering the AWS Glue database and tables approved for access. These consumers can now query the datasets using the AWS analytics services of their choice like Amazon Redshift Spectrum, Amazon Athena, and Amazon EMR.

Operational excellence

Along with building the data mesh, it’s also important to verify that we can operate with efficiency and reliability. We recognize that the metadata sync process is at the heart of this global data catalog. As such, we are hypervigilant of this process and have built alarms, notifications, and dashboards to verify that this process doesn’t fail silently and create a single point of failure for the global data catalog. We also have a backup repair service that syncs the metadata from producer catalogs into the central governance account catalog periodically. This is a self-healing mechanism to maintain reliability and resiliency.

Empowering customers with the data mesh

The FinAuto data mesh hosts around 850 discoverable and shareable datasets from multiple partner accounts. There are more than 300 curated data products to which producers can provide access and apply governance with fine-grained access controls. Our consumers use AWS analytics services such as Redshift Spectrum, Athena, Amazon EMR, and Amazon QuickSight to access their data. This capability with standardized data vending from the data mesh, along with self-serve capabilities, allows you to innovate faster without dependency on technical teams. You can now get access to data faster with automation that continuously improves the process.

By serving the FinOps team’s data needs with high availability and security, we enabled them to effectively support operation and reporting. Data science teams can now use the data mesh for their finance-related AI/ML use cases such as fraud detection, credit risk modeling, and account grouping. Our finance operations analysts are now enabled to dive deep into their customer issues, which is most important to them.

Conclusion

FinOps implemented a data mesh architecture with Lake Formation to improve data governance with fine-grained access controls. With these improvements, the FinOps team is now able to innovate faster with access to the right data at the right time in a self-serve manner to drive business outcomes. The FinOps team will continue to innovate in this space with AWS services to further provide for customer needs.

To learn more about how to use Lake Formation to build a data mesh architecture, see Design a data mesh architecture using AWS Lake Formation and AWS Glue.


About the Authors

Nitin Arora PicNitin Arora is a Sr. Software Development Manager for Finance Automation in Amazon. He has over 18 years of experience building business critical, scalable, high-performance software. Nitin leads several data and analytics initiatives within Finance, which includes building Data Mesh. In his spare time, he enjoys listening to music and read.

Pradeep Misra PicPradeep Misra is a Specialist Solutions Architect at AWS. He works across Amazon to architect and design modern distributed analytics and AI/ML platform solutions. He is passionate about solving customer challenges using data, analytics, and AI/ML. Outside of work, Pradeep likes exploring new places, trying new cuisines, and playing board games with his family. He also likes doing science experiments with his daughters.

Rajesh Rao PicRajesh Rao is a Sr. Technical Program Manager in Amazon Finance. He works with Data Services teams within Amazon to build and deliver data processing and data analytics solutions for Financial Operations teams. He is passionate about delivering innovative and optimal solutions using AWS to enable data-driven business outcomes for his customers.

Andrew Long PicAndrew Long, the lead developer for data mesh, has designed and built many of the big data processing systems that have fueled Amazon’s financial data processing infrastructure. His work encompasses a range of areas, including S3-based table formats for Spark, diverse Spark performance optimizations, distributed orchestration engines and the development of data cataloging systems. Additionally, Andrew finds pleasure in sharing his knowledge of partner acrobatics.

Satyen GauravKumar Satyen Gaurav, is an experienced Software Development Manager at Amazon, with over 16 years of expertise in big data analytics and software development. He leads a team of engineers to build products and services using AWS big data technologies, for providing key business insights for Amazon Finance Operations across diverse business verticals. Beyond work, he finds joy in reading, traveling and learning strategic challenges of chess.

The collective thoughts of the interwebz