Test Methodology
Principles
- Reproducible before report: a test that can't be re-run doesn't enter the public record
- External ground truth: for each protocol we also verify with a third-party client (
curl,mc,ssh,openssl) to distinguish AeroFTP regressions from server misconfig - Integrity check: every heavy upload/download is verified with SHA-256
- Exit code first: every command must terminate with the correct exit code; textual output is secondary
Docker harness
Full local environment, containers exposed on localhost only. The files are published in the docs repository under tools/docker-harness, outside the site's public tree so that generated keys can never be copied into a site build: the compose file, the SFTP image and setup.sh, which generates the SSH test keys on your machine. No key is committed, and the credentials below are for these local containers only.
| Service | Host port | Protocol | Credentials |
|---|---|---|---|
aeroftp-test-ftps | 2121 | FTP (vsftpd), plain FTP without TLS | ftpuser / password123 |
aeroftp-test-sftp | 2223 | SFTP (OpenSSH) | user_key with key, user_pwd with password, user_mixed with key and password |
aeroftp-test-webdav | 8080 | WebDAV (bytemark/webdav) | webdavuser / password123 |
aeroftp-test-minio | 9000 / 9001 | S3 (MinIO) | admin / password123 |
Quick start, from the docker-harness directory:
./setup.sh
docker compose up -d --buildInitial S3 bucket
mc alias set test http://127.0.0.1:9000 admin password123
mc mb --ignore-existing test/aeroftp-testCoverage Class and scoring
The Provider Coverage Matrix assigns each provider a Coverage Class (A, B, C, D) and a numeric score out of 100 derived from a deterministic, speed-independent rubric.
| Class | Label | Score | Meaning |
|---|---|---|---|
| A | Primary | 90-100 | Full matrix green, ready for critical workloads |
| B | Extended | 70-89 | Core operations solid, minor gaps on advanced features |
| C | Compatible | 50-69 | Base works, known non-blocking limitations |
| D | Observer | < 50 | Partially covered, not recommended for production |
Rubric (100 pt total):
| Dimension | Weight | What it measures |
|---|---|---|
| Core Operations | 30 | connect, ls, put, get, stat, mkdir, rm, mv |
| Data Integrity | 20 | SHA-256 end-to-end + hashsum parity |
| Navigation & Discovery | 15 | tree, find, head, cat, recursive stat |
| Advanced Features | 15 | share links, trash/restore, versions, server-side copy, quota |
| Encoding Robustness | 10 | unicode, spaces, special characters in file names |
| Reconciliation | 10 | check, sync-doctor, reconcile post-sync matches |
Throughput is deliberately excluded from scoring: it depends on distance from the provider endpoint and local network conditions, not on the client implementation.
Matrix conventions
Symbols used in tables:
- ✅ passes completely with verified integrity
- ⚠️ passes with caveat (noted below the table)
- ⏳ pending benchmark (dimension not yet consolidated for this provider)
- ❌ fails
- "-" not applicable / not tested
Matrices are plain HTML tables without custom styling. The test-reports section uses a full-width layout to host them even when the grid is dense.
Comparative benchmarks
The DAG engine review and the comparative battery measure AeroFTP against other clients on the same link. They follow these rules:
- The same cell, the same window. Each tool runs the same payload against the same server back to back, with the other tools in the same operation. The tool order is reversed between passes, so a drift over time does not always land on the same tool.
- A control on every row. rclone runs every cell. It is evidence about shared drift, not proof of cause: when AeroFTP moves between two builds and rclone does not, the change most likely belongs to the build, and when both move, the window most likely does. A change in the network or the server can still affect two clients differently.
- Integrity before speed. Every download is compared with the source by SHA-256, file by file for trees. A timing without that check is not published: a failed or truncated transfer can look fast. Exit codes are recorded but not trusted alone, since a client can exit 0 after a partial transfer.
- Pairs, not single numbers. Rows show each run or the spread. A single run is labelled as one run. The noise floor is measured per target and per tool, using the tool that does not change between arms.
- Ratios, not absolute speed. Ratios are the other tool's seconds over AeroFTP's seconds. The link is the bottleneck for every tool, so a ratio travels better than a throughput figure.
- Pinned binaries. Each AeroFTP build is named by commit and sha256, and checked by probing its behaviour, not by the directory it came from. The versions of the other tools are recorded with each session.
- One configuration source. The rclone and Cyberduck configurations are exported from the same AeroFTP vault with
aeroftp-cli export rcloneandaeroftp-cli export cyberduck, so every tool talks to the same endpoint with the same credentials and TLS mode. - Asymmetric cells are labelled or dropped. TLS against cleartext, different stream counts or different payloads make a row incomparable; such rows carry the clause or are left out.
- Anomalies are repeated before they are reported. A finding that does not survive repetition with the control is left out, and the page says what was left out and why.
What these measurements do not establish: a ranking of clients in general. They describe one station, one link and one set of servers.
The comparative harness itself is a maintainer script bound to the lab profiles and is not published. The rules above, the commands listed on each page and aeroftp-cli export are enough to rebuild an equivalent battery against your own servers.
What this suite is not
This suite is not:
- a rigorous performance benchmark across networks (the comparative pages above measure one link, with the method described in Comparative benchmarks)
- a security certification (that lives in Security)
- an API contract (that lives in CLI and MCP)
It's an operational regression-testing log, published for transparency.