vrr/dpi-checkers
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add tcp-16-20 dwc checker

Browse source
This commit is contained in:
Petr Osetrov 2025-07-02 22:54:53 +03:00
parent c32059ff4c
commit a3a16be400
5 changed files with 450 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

5
README.md
View file

@ -1,5 +1,5 @@
# DPI Checkers
🚀 This repository contains checkers that allow you to determine if your “home” ISP has DPI, as well as the specific methods the censor uses for limitations.
🚀 This repository contains checkers that allow you to determine if your “home” ISP has DPI, as well as the specific methods (and their parameters) the censor uses for limitations.
## Checkers list
- **RU :: TCP 16-20** => [https://hyperion-cs.github.io/dpi-checkers/ru/tcp-16-20](https://hyperion-cs.github.io/dpi-checkers/ru/tcp-16-20)<br>
@ -13,6 +13,9 @@
| provider | string | _Custom_ | Provider name (you can set any name). |
See [here](https://github.com/net4people/bbs/issues/490) for details on this blocking method.
- **RU :: TCP 16-20 DWC** (domain whitelist checker)<br>
Allows to find out whitelisted items on DPIs where _TCP 16-20_ blocking method is applied. This kind of information can be interesting in its own right as well as useful for bypassing limitations.<br>
A list of domains is required as input. Also requires _Python 3_, the _curl_ utility, and a specially configured server on "limited" networks. See [here](ru/tcp-16-20_dwc) for details (ready-to-use results are also available for download there).
## Contributing
We would be happy if you could help us improve our checkers through PR or by creating issues.

59
ru/tcp-16-20_dwc/README.md Normal file
View file

@ -0,0 +1,59 @@
# RU :: TCP 16-20 DWC (domain whitelist checker)
Allows to find out whitelisted items on DPIs where _TCP 16-20_ blocking method is applied. This kind of information can be interesting in its own right as well as useful for bypassing limitations.<br>
A list of domains is required as input. Also requires _Python 3_, the _curl_ utility, and a specially configured server on "limited" networks. See belowe for details.
## Ready-to-use results
Not everyone will want to run this script on their own (especially because it can run for quite a long time, and because its implementation is naive and uses a bruteforce method). That's why this work has already been done by the committers of this repository.
The top-10k popular domains based on the list from [OpenDNS](https://github.com/opendns/public-domain-lists/blob/master/opendns-top-domains.txt) (unfortunately, this file was last updated _2014-11-06_, but it's still generally up to date) was used as a input list.
**Last Updated**: _2025-07-02_<br>
**Last File**: [/ru/tcp-16-20_dwc/results/based_on_opendns_2025-07-02.txt](/ru/tcp-16-20_dwc/results/based_on_opendns_2025-07-02.txt)<br>
**Latest stats**: _266_ domains out of _10'000_ (_2.66%_) are whitelisted<br>
**File Format**: _.csv/.md_ table with `| Domain | Provider | Country |` header
### Notes
As far as we know, the whitelist is created using the `*.domain.com:*` scheme. Thus, you can (and should?) use subdomains of the found domains (if _site.com_ works, then _foo.site.com_ and _foo.bar.site.com_ will also work).
We also bring to your attention a graph that shows the dependence of being on the whitelist on the place in the top (provided by OpenDNS).
![graph](https://github.com/user-attachments/assets/7fcdd150-99ce-4686-9c11-d0fab2610697)
It can be seen that there is a correlation between these properties (which is generally logical).
## Self-running the script
1. First of all, you have to get an input list with domains to test (the script doesn't get the full whitelist, it just checks your input list to see if each of its elements is included in the DPI whitelist). You can use OpenDNS lists as a starting point (see above);
2. You will need a remote server in “suspicious” networks (i.e. those limited by _TCP 16-20_ blocking method at your “home” ISP). There, you would need to install a web server with https (a self-signed certificate [_openssl_/etc] is fine, since the script ignores validation) that would respond the same regardless of the SNI passed. It should also send a file of at least 128KB (over the network, including compression) to some path — GET request.<br>
As such a server you can use _nginx_ with approximately the following configuration:
```nginx
server {
listen 443 ssl default_server;
ssl_certificate /path/to/cert.crt;
ssl_certificate_key /path/to/cert.key;
root /var/www/html;
location / {
try_files $uri $uri/ =404;
}
}
```
A static file can be generated like this (in this case, 1MB in size):
```bash
dd if=/dev/urandom of=/var/www/html/1MB.bin bs=1M count=1
```
\* Don't forget to open https (443) port.
3. Finally, on your local machine (must have Python 3 and the `curl` utility installed) with internet access through an ISP with DPI using the TCP 16-20 blocking method, you can run the script. It is recommended to use a POSIX-compatible OS (Linix, macOS, etc). The script has the following parameters:
| Parametr | Default | Required | Desc |
| :-: | :-: | :-: | - |
| `-i` | _in.txt_ | No |Path to the file with the list of domains to check.|
| `-o` | _out.txt_ | No |The path to the results file. The domains that are included in the whitelist will be saved.|
| `-e` | _err.txt_ | No |Error file path.|
| `-u` | — | Yes |The path for the URL where the static file is located.|
| `-d` | — | Yes |IP of your destination server from the previous step.|
| `-t` | `5` | No |Connection/read timeout in seconds.|
| `-r` | `65535` | No |Upper bound of the range of bytes to be downloaded.|
Example of a run:
```bash
python domain_whitelist_checker.py -u /1MB.bin -d 1.2.3.4
```
The script is single-threaded, but you can parallelize it via e.g. GNU [parallel](https://www.gnu.org/software/parallel/) utility.
Also you can run the result file through [this](/utils/domain2provider.py) script to find out the likely ISPs the domain owners are using, as well as the country.

52
ru/tcp-16-20_dwc/domain_whitelist_checker.py Normal file
View file

@ -0,0 +1,52 @@
import subprocess
import datetime
import argparse
def log(msg):
ts = datetime.datetime.now().strftime("%H:%M:%S.%f")[:-3]
print(f"[{ts}] {msg}")
def main():
parser = argparse.ArgumentParser(prog="Domain whitelist checker")
parser.add_argument("-i", dest="in_path", default="in.txt", help="Input file path")
parser.add_argument("-o", dest="out_path", default="out.txt", help="Output file path")
parser.add_argument("-e", dest="err_path", default="err.txt", help="Error file path")
parser.add_argument("-u", dest="url_path", required=True, help="URL path")
parser.add_argument("-d", dest="dst_ip", required=True, help="Destination IP")
parser.add_argument("-t", dest="timeout_sec", type=int, default=5, help="Connection/read timeout in seconds")
parser.add_argument("-r", dest="range_to", type=int, default=65535, help="Upper bound of the range of bytes to be downloaded")
args = parser.parse_args()
with open(args.in_path, "r") as infd:
items = [line.strip() for line in infd if line.strip()]
total = len(items)
with open(args.out_path, "w") as outfd, open(args.err_path, "w") as errfd:
for idx, domain in enumerate(items, start=1):
try:
result = subprocess.getoutput(
f'curl -k https://{domain}{args.url_path} \
--resolve {domain}:443:{args.dst_ip} \
-o /dev/null \
-s -w "%{{size_download}}\n" \
--max-time {args.timeout_sec} \
--range 0-{args.range_to}'
)
bytes = float(result)
if bytes >= args.range_to:
outfd.write(domain + "\n")
log(f"[{idx}/{total}] {domain} => {bytes}")
except Exception as e:
errfd.write(domain + "\n")
log(f"[{idx}/{total}] {domain} => ERROR: {e}")
if __name__ == "__main__":
main()

267
ru/tcp-16-20_dwc/results/based_on_opendns_2025-07-02.txt Normal file
View file

@ -0,0 +1,267 @@
|Domain|Provider|Country|
|9gag.com|Cloudflare, Inc.|US|
|academia.edu|Amazon Technologies Inc.|US|
|accuweather.com|Akamai Technologies, Inc.|US|
|acer.com|Amazon.com, Inc.|US|
|acint.net|RIPE Network Coordination Centre|NL|
|adgrx.com|—|—|
|admost.com|CloudFlare, Inc.|US|
|adobe.com|MTS PJSC|RU|
|adobedtm.com|—|—|
|adobelogin.com|Adobe Systems Incorporated|US|
|adweek.com|Automattic, Inc|US|
|alibaba.com|Alibaba Cloud LLC|US|
|alicdn.com|Asia Pacific Network Information Centre|AU|
|aliexpress.com|Alibaba Cloud LLC|US|
|aliyun.com|Zhejiang Taobao Network Co.,Ltd|CN|
|aliyuncs.com|Aliyun Computing Co., LTD|CN|
|amazon-adsystem.com|Amazon Technologies Inc.|US|
|amazon.co.uk|Amazon Technologies Inc.|US|
|amazon.com|Amazon Technologies Inc.|US|
|amd.com|Akamai Technologies|EU|
|apache.org|Fastly, Inc.|US|
|apple.com|Apple Inc.|US|
|appsflyer.com|Amazon Technologies Inc.|US|
|arcgis.com|Amazon.com, Inc.|US|
|asana.com|Amazon Technologies Inc.|US|
|askubuntu.com|CloudFlare, Inc.|US|
|asus.com|ASUSTek COMPUTER INC.|TW|
|atlassian.com|Amazon Technologies Inc.|US|
|audible.com|Amazon Technologies Inc.|US|
|avito.ru|KEH eCommerce LLC|RU|
|basecamp.com|CloudFlare, Inc.|US|
|battle.net|RIPE Network Coordination Centre|NL|
|battlefield.com|Akamai Technologies, Inc.|US|
|behance.net|Fastly, Inc.|US|
|bing.com|Microsoft Corporation|AU|
|bitbucket.org|eu-central-1|DE|
|blizzard.com|RIPE Network Coordination Centre|NL|
|bluestacks.com|Amazon Technologies Inc.|US|
|booking.com|Amazon Technologies Inc.|US|
|bootstrapcdn.com|Render|US|
|box.com|Box.com|US|
|boxcdn.net|—|—|
|bpsecure.com|Bigpoint GmbH|DE|
|britannica.com|Amazon Technologies Inc.|US|
|bstatic.com|Amazon.com, Inc.|US|
|bungie.net|Cloudflare, Inc.|US|
|callofduty.com|Akamai Technologies, Inc.|US|
|cambridge.org|Cloudflare, Inc.|US|
|cdnjs.com|Cloudflare, Inc.|US|
|centos.org|Eweka Internet Services B.V.|NL|
|change.org|Cloudflare, Inc.|US|
|cisco.com|CISCO SYSTEMS, INC.|US|
|cnn.com|Fastly, Inc.|US|
|codecanyon.net|CloudFlare, Inc.|US|
|counter-strike.net|Akamai Technologies, Inc.|US|
|crwdcntrl.net|Lotame Solutions, Inc.|US|
|curseforge.com|Cloudflare, Inc.|US|
|dailymail.co.uk|Akamai Technologies|EU|
|deezer.com|Amazon Technologies Inc.|US|
|demdex.net|—|—|
|deviantart.com|Amazon.com, Inc.|US|
|discogs.com|Amazon Technologies Inc.|US|
|dota2.com|CloudFlare, Inc.|US|
|dribbble.com|Amazon Technologies Inc.|US|
|dropbox.com|Dropbox, Inc.|US|
|dropboxstatic.com|Dropbox, Inc.|US|
|drweb.com|OOO "Doktor Veb"|RU|
|duolingo.com|Amazon.com, Inc.|US|
|ea.com|Akamai Technologies|EU|
|ebay.com|Akamai Technologies, Inc.|US|
|edgekey.net|—|—|
|envato.com|Cloudflare, Inc.|US|
|eset.com|ESET, spol. s r.o.|SK|
|esoui.com|Cloudflare, Inc.|US|
|fastclick.net|—|—|
|fastpic.ru|OVH SAS|FR|
|fedoraproject.org|Amazon Technologies Inc.|US|
|fifa.com|Akamai Technologies|EU|
|flickr.com|Amazon Technologies Inc.|US|
|freshdesk.com|Amazon Technologies Inc.|US|
|fsdn.com|Internet Express|US|
|gameanalytics.com|Amazon.com, Inc.|US|
|gameforge.com|Cloudflare, Inc.|US|
|gameloft.com|Divertissements GameLoft Inc|CA|
|garmin.com|Cloudflare, Inc.|US|
|genius.com|CloudFlare, Inc.|US|
|getbootstrap.com|CloudFlare, Inc.|US|
|gismeteo.ru|"MapMakers Group" Ltd|RU|
|github.com|GitHub, Inc.|US|
|githubusercontent.com|—|—|
|globalsign.com|Cloudflare, Inc.|US|
|go-mpulse.net|—|—|
|goodgamestudios.com|Amazon Technologies Inc.|US|
|googleapis.com|Google LLC|US|
|googletagmanager.com|Google LLC|US|
|gsmarena.com|RIPE Network Coordination Centre|NL|
|helpshift.com|WPEngine, Inc.|US|
|hm.com|Akamai Technologies|EU|
|hotjar.com|Amazon Technologies Inc.|US|
|hp.com|HP Inc.|US|
|hubspot.com|Cloudflare, Inc.|US|
|ibm.com|Akamai Technologies, Inc.|US|
|iconfinder.com|CloudFlare, Inc.|US|
|ieee.org|Institute of Electrical and Electronics Engineers, Inc|US|
|ietf.org|Cloudflare, Inc.|US|
|igg.com|Amazon Technologies Inc.|US|
|imdb.com|Amazon Technologies Inc.|US|
|imvu.com|IMVU, Inc|US|
|inmobi.com|Microsoft Corporation|US|
|inn.ru|Innova RU Infrastructure|RU|
|intel.com|Microsoft Corporation|US|
|internetat.tv|Asia Pacific Network Information Centre|AU|
|iobit.com|Amazon Technologies Inc.|US|
|itunes.com|Apple Inc.|US|
|jetbrains.com|Amazon Technologies Inc.|US|
|jsdelivr.net|Resource Quality Assurance|AU|
|jtvnw.net|—|—|
|kaspersky-labs.com|—|—|
|kaspersky.com|Kaspersky Lab Switzerland GmbH|RU|
|kia.com|Kia America, Inc.|US|
|lastpass.com|Akamai Technologies|EU|
|leagueoflegends.com|Amazon Technologies Inc.|US|
|lg.com|Amazon Technologies Inc.|US|
|libreoffice.org|Hetzner Online GmbH|DE|
|licdn.com|—|—|
|live.com|Microsoft Corporation|US|
|mail.ru|VK Services|RU|
|mapbox.com|Fastly, Inc.|US|
|marriott.com|Akamai Technologies, Inc.|US|
|mayoclinic.org|Mayo Foundation for Medical Education and Research|US|
|mediafire.com|Cloudflare, Inc.|US|
|merriam-webster.com|Amazon Technologies Inc.|US|
|microsoft.com|Microsoft Corporation|US|
|minecraft.net|Microsoft Corporation|US|
|mit.edu|Akamai Technologies, Inc.|US|
|miui.com|21ViaNet(China),Inc.|CN|
|mobile.de|Amazon Technologies Inc.|US|
|mojang.com|Microsoft Corporation|US|
|msecnd.net|—|—|
|msn.com|Microsoft Corporation|US|
|myfitnesspal.com|CloudFlare, Inc.|US|
|mysql.com|Oracle Corporation|US|
|netflix.com|Amazon Technologies Inc.|US|
|nginx.org|Amazon Technologies Inc.|US|
|nhl.com|CloudFlare, Inc.|US|
|nintendo.com|Nintendo Of America inc.|US|
|nintendo.net|—|—|
|nist.gov|Cloudflare, Inc.|US|
|nvidia.com|Amazon Technologies Inc.|US|
|office.com|Microsoft Corporation|US|
|office365.com|Microsoft Corporation|US|
|ok.ru|Odnoklassniki Services|RU|
|omtrdc.net|—|—|
|openstreetmap.org|Cloudflare, Inc.|US|
|opera.com|Opera Software AS|US|
|optimizely.com|EPiServer Hosting SE|SE|
|origin.com|Akamai Technologies, Inc.|US|
|pastebin.com|CloudFlare, Inc.|US|
|pepsico.com|Incapsula Inc|US|
|php.net|Myra Security GmbH|DE|
|pinimg.com|Fastly, Inc.|US|
|pinterest.com|Fastly, Inc.|US|
|pixlr.com|Amazon Technologies Inc.|US|
|playstation.net|—|—|
|playwire.com|Amazon Technologies Inc.|US|
|plex.tv|Amazon Technologies Inc.|US|
|pravda.ru|CloudFlare, Inc.|US|
|psychologytoday.com|Amazon Technologies Inc.|US|
|pubmatic.com|Amazon Technologies Inc.|US|
|pushwoosh.com|Hetzner Online GmbH|FI|
|pvp.net|—|—|
|quickconnect.to|Amazon Technologies Inc.|US|
|quizlet.com|Cloudflare, Inc.|US|
|rambler.ru|Rambler Head|RU|
|rarlab.com|Canboy Burak|DE|
|rbc.ru|AO <<ROSBIZNESKONSALTING>>|RU|
|redhat.com|Amazon Technologies Inc.|US|
|researchgate.net|CloudFlare, Inc.|US|
|reuters.com|Thomson Reuters U.S. LLC|US|
|reverso.net|CloudFlare, Inc.|US|
|riotgames.com|Akamai Technologies, Inc.|US|
|roblox.com|Roblox|US|
|rockstargames.com|Akamai Technologies|EU|
|rutarget.ru|"Cloud Technologies" LLC trading as Cloud.ru|RU|
|samsung.com|SamsungSDS Inc.|KR|
|samsungapps.com|Samsung SDS Europe Ltd. German Branch|DE|
|samsungcloudsolution.com|—|—|
|samsungcloudsolution.net|—|—|
|samsungdm.com|—|—|
|samsungosp.com|—|—|
|samsungotn.net|SamsungSDS Inc.|KR|
|savefrom.net|CloudFlare, Inc.|US|
|sciencedirect.com|Elsevier Limited|GB|
|scorecardresearch.com|CenturyLink Communications, LLC|US|
|sendgrid.com|Amazon Technologies Inc.|US|
|serverfault.com|CloudFlare, Inc.|US|
|shopify.com|Shopify, Inc.|CA|
|shutterstock.com|Amazon.com, Inc.|US|
|skype.com|Microsoft Corporation|US|
|slack.com|Amazon Technologies Inc.|US|
|softpedia.com|Aptum Technologies|CA|
|sonos.com|Akamai International BV|FI|
|sony.com|Amazon Technologies Inc.|US|
|sony.tv|—|—|
|sonyentertainmentnetwork.com|Oracle Corporation|US|
|sourceforge.net|Cloudflare, Inc.|US|
|spanishdict.com|CloudFlare, Inc.|US|
|speedtest.net|Fastly, Inc.|US|
|spotify.com|Google LLC|US|
|ssl-images-amazon.com|—|—|
|stackexchange.com|CloudFlare, Inc.|US|
|stackoverflow.com|CloudFlare, Inc.|US|
|staticflickr.com|—|—|
|statuspage.io|Atlassian Network Services, Inc.|US|
|steamcommunity.com|Akamai Technologies|EU|
|steampowered.com|Akamai Technologies, Inc.|US|
|steamstatic.com|—|—|
|sublimetext.com|DigitalOcean, LLC|US|
|superuser.com|CloudFlare, Inc.|US|
|synology.com|Amazon.com, Inc.|US|
|tango.me|Google LLC|US|
|taobao.com|Hangzhou Alibaba Advertising Co.,Ltd.|CN|
|teamspeak.com|CloudFlare, Inc.|US|
|timeanddate.com|Cloudflare, Inc.|US|
|tnt-ea.com|—|—|
|todoist.com|Amazon.com, Inc.|US|
|tp-link.com|Amazon.com, Inc.|US|
|trello.com|Amazon Technologies Inc.|US|
|tripadvisor.com|Fastly, Inc.|US|
|truste.com|Amazon.com, Inc.|US|
|twitch.tv|Fastly, Inc.|US|
|udemy.com|Cloudflare, Inc.|US|
|uefa.com|Microsoft Corporation|US|
|unica.com|Corporation Service Company|US|
|unity3d.com|Akamai Technologies, Inc.|US|
|usbank.com|U.S. BANCORP|US|
|valvesoftware.com|Akamai Technologies, Inc.|US|
|videolan.org|Free Foundation (free.org)|FR|
|vimeo.com|Cloudflare, Inc.|US|
|visualstudio.com|Microsoft Corporation|AU|
|vk.me|VKontakte Services|RU|
|vkontakte.ru|VKontakte Services|RU|
|vmware.com|Amazon.com, Inc.|US|
|vungle.com|WPEngine, Inc.|US|
|w3.org|CloudFlare, Inc.|US|
|weather.com|Akamai Technologies|EU|
|webex.com|Cisco Webex LLC|US|
|webmoney.ru|RIPE Network Coordination Centre|NL|
|weibo.com|15F,Ideal Plaza No.58 Bei Si Huan Xi Road Haidian District|CN|
|whatsapp.com|Facebook, Inc.|US|
|wikipedia.org|Wikimedia esams infra|NL|
|wiley.com|Verizon Business|US|
|windows.com|Microsoft Corporation|US|
|worldoftanks.com|G-Core Labs S.A.|NL|
|wowhead.com|Amazon Technologies Inc.|US|
|xbox.com|Microsoft Corporation|US|
|xboxlive.com|Microsoft Corporation|US|
|xiaomi.com|21ViaNet(China),Inc.|CN|
|yahoo.com|Oath Holdings Inc.|US|
|yandex.com|YANDEX LLC|RU|
|yandex.kz|YANDEX LLC|KZ|
|yandex.net|YANDEX LLC|RU|
|youtube.com|Google LLC|US|
|zamimg.com|—|—|
|zara.com|Akamai International BV|FI|
|zoho.com|NTT America, Inc.|US|

68
utils/domain2provider.py Normal file
View file

@ -0,0 +1,68 @@
import socket
import subprocess
import re
def extract_operator(whois):
provider_fields = [
"OrgName",
"Organization",
"org-name",
"descr",
"owner",
"netname",
"responsible",
"role",
"person",
"mnt-by",
]
for field in provider_fields:
match = re.search(rf"^{field}\s*:\s*(.+)$", whois, re.IGNORECASE | re.MULTILINE)
if match:
return match.group(1).strip()
return "—"
def extract_country(whois):
country_fields = ["country", "ctry", "co", "country-code"]
for field in country_fields:
match = re.search(
rf"^{field}\s*:\s*(\w+)$", whois, re.IGNORECASE | re.MULTILINE
)
if match:
return match.group(1).strip().upper()
return "—"
def main():
with open("in.txt", "r") as infd:
domains = [line.strip() for line in infd if line.strip()]
total = len(domains)
with open("out.txt", "w") as outfd:
outfd.write(f"|Domain|Provider|Country|\n")
for idx, domain in enumerate(domains, start=1):
provider = "—"
country = "—"
try:
ip = socket.gethostbyname(domain)
result = subprocess.run(
f"whois {ip}",
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
text=True,
)
provider = extract_operator(result.stdout)
country = extract_country(result.stdout)
except Exception:
pass
print(f"{idx}/{total}: {domain} => {provider} [{country}]")
outfd.write(f"|{domain}|{provider}|{country}|\n")
if __name__ == "__main__":
main()