• Test Reports
• ›
• DAG engine review, September 2026

DAG engine review, September 2026 ​

This is the record of a review of the DAG transfer engine in September 2026. The review measured each fix against the build before it, with rclone running the same cell in the same window as a control. It is published as a dated record: the builds are review builds, not releases, and the current release is measured in the comparative battery of October 2026.

How to read these numbers ​

• The control is what makes a row readable. Contention on the station or on the shared uplink usually moves both tools in a cell. A row where AeroFTP moved and rclone did not most likely describes the code, and a row where both moved most likely describes the window; it is evidence, not proof, since a network or server change can affect two clients differently.
• The noise floor is per target and per tool. In a quiet window AeroFTP rows spread 2 to 5 percent on S3 and WebDAV; one rclone pair on WebDAV spread 17.5 percent; FTP reached 42 percent between identical runs. A single run on this bench cannot separate a 40 percent effect from noise, so every table says how many runs it has.
• Ratios are rclone seconds over AeroFTP seconds. Above 1.00, AeroFTP is faster. The link is the bottleneck for both tools, so ratios travel better than absolute seconds.
• FTP rows are not like-for-like unless stated. The FTP lab profile uses explicit TLS (the default for a new FTP profile in AeroFTP), while the rclone remote in this battery used plain FTP. That asymmetry penalises AeroFTP. The one like-for-like FTP cell is marked as such.

Setup ​

• Station: one Linux desktop, 12 cores, wired gigabit. Another job compiled Rust on the same station during both main sessions, and a second machine shared the uplink. Both confounds touch absolute seconds; neither can make one tool move while the other stays still.
• Targets: one remote lab server running OpenSSH (SFTP), vsftpd (FTP with explicit TLS), Apache mod_dav behind nginx (WebDAV) and MinIO (S3).
• Payloads: 5,000 files of 4 KiB ("small tree"), one 300 MiB file, a 200-file mixed tree, and 20 MiB under --limit-rate 2M for the rate cap cells.
• Integrity: every download was compared with the source by SHA-256, file by file for trees. A timing without that check is not reported.
• Binaries: each build was pinned by commit and identified by probing its behaviour, not by the directory it came from.

1. Session reuse on SFTP, and sync on S3 ​

One run per cell, 8 September 2026. Seconds, lower is better.

rclone stayed within 9 percent on every control while AeroFTP moved by 47 to 95 percent, so the movement belongs to the build. The --parallel 16 rows did not improve over --parallel 4 because on these builds the SFTP pool stopped at four sessions; section 3 shows what happened when that ceiling was raised. The 300 MiB SFTP rows are single-stream on this build; the channel effect on them is on the fixed pipe page.

2. The rate cap: slower is the correct answer ​

20 MiB under --limit-rate 2M should take 10.0 seconds. A faster time means the cap was ignored. One run per cell.

A table of percentage changes without the 10-second target would show these rows as a large regression.

The FTP upload under the cap stayed about 3 seconds over target after #735. The overshoot was constant in seconds (2.93 s over a 10 s target, 3.03 s over a 20 s one), which pointed at a fixed cost rather than a mistuned limiter: the client waited a timed interval before closing each data connection. #769 waits for the server's signal instead. Two runs, prediction written into the script before measuring:

3. SFTP above four sessions ​

Before #761, the SFTP provider capped its pool at four sessions, so --parallel 16 did what --parallel 4 did. Two runs per build, interleaved, integrity verified.

The --parallel 16 rows halve and the --parallel 4 rows do not move: the ceiling was raised without changing the default. A socket count taken during the runs read 10 connections at --parallel 16 and 6 at --parallel 4 on the new build (the count includes the main connection), so the pool follows the request instead of always opening its maximum.

Against rclone at --transfers 16, after this fix: upload 145.99 s against 344.00 s (ratio 2.36), download 126.47 s against 109.89 s (ratio 0.87).

4. Closing the last SFTP gap: stat and open overlapped ​

The download at --parallel 16 was the last row of the review where rclone was ahead. #782 overlaps the stat and open round trips for small files whose size is already known from the listing. Two interleaved runs, 20 integrity checks, 0 failed.

The medians move from 135.06 s to 101.54 s (-24.8 percent) while rclone moves by +0.5 percent. rclone in the same runs sits at about 107.5 s, so AeroFTP finished this cell about 5.6 percent ahead (ratio 1.06). The same change also improved --parallel 4 from 252.17 s to 202.04 s, where rclone took 382.62 s in the same window (one run per build, ratio 1.89).

The saving was larger than a simple model predicts: one round trip per file over ten sessions accounts for about 25 seconds, and the measured saving was 33.5 seconds. The direction and the order of magnitude agree; the remaining third is not explained here.

5. A regression found and fixed in the same review ​

The "after" build added one size probe before every small-file download (a signed HEAD on S3, a one-byte Range request on WebDAV). Three builds in one window, interleaved twice, integrity verified on all twelve runs:

The added time matches one extra round trip per file on each of the four download channels: 5,000 files over 4 channels is 1,250 files per channel, and the mean increases (58.0 s on S3, 56.7 s on WebDAV) divided by 1,250 give 46 ms and 45 ms, against a measured RTT of 47.36 ms. #757 removed the probe.

6. check on a populated remote ​

#758 moved the remote tree walk of check, reconcile and cryptcheck onto parallel sessions. One run per build: check opened 5 sessions instead of 1 and went from 11.37 s to 5.78 s. sync reached the remote tree through other walkers on that build and did not change; #762 followed for it.

7. FTP, like-for-like ​

The only FTP cell of the review where both tools ran the same transport. 1 MiB upload, three runs per arm, base commit 43a37db1c.

Plain against plain the two tools are level. With TLS, AeroFTP is faster. Against this server (vsftpd 3.0.5, which requires each data connection to reuse the control connection's TLS session), rclone v1.74.0 with its default settings did not complete an upload over explicit TLS, and with --ftp-disable-tls13 it did; the mechanism on rclone's side was not isolated, so this is a result for this server and these versions. AeroFTP caps FTP over TLS at TLS 1.2 because, with its TLS library, a TLS 1.3 session ticket is consumed when used and a second data connection would not resume the control connection's session; its rclone exporter writes disable_tls13 into exported FTP remotes since 4.2.0.

An earlier reading of this cell compared AeroFTP over TLS with rclone in plain FTP and reported a 0.48 s gap. That arm of the cell had failed on every run and was not counted; the figure above replaces it.

8. What was left out, and why ​

The first pass of the review flagged eight anomalies. Each was repeated with the rclone control before anything was concluded.

Six of the eight did not survive repetition, and the seventh was a real cost with a different cause. Publishing after the first pass would have reported six false leads.

9. The channel sweep, and one row that does not measure what it says ​

The review also swept the number of channels on one 300 MiB download per protocol. Those tables live on the fixed pipe page, because they measure channel count, not the DAG engine against an older path.

The S3 points of that sweep were all single-stream. On these builds the S3 multi-stream download took the object size from a HEAD response through a call that always returns 0 for HEAD, so no download ever reached the range path, whatever the requested channel count. #881 fixed it on 19 September. The differences between the S3 points are noise, and the sweep says nothing about S3 ranged downloads. They are measured on 4.2.1 in the comparative battery.

Declared limits ​

• One station, one uplink, one lab server. These numbers describe this link and these servers, not a general ranking.
• Rows marked as one run are one run. On this bench a single run cannot separate a 40 percent effect from noise.
• The review builds sit between 4.1.9 and 4.2.0; the fixes listed above shipped in 4.2.0.
• The FTP rows outside section 7 compare AeroFTP over explicit TLS with rclone in plain FTP.
• The station's CPU was shared with a compiler during both main sessions, and the uplink with a second machine. Both affect absolute seconds for both tools.