Readme update

README.md

@@ -1,6 +1,6 @@
 # sudofox/mojang-blocklist
 
-In September of 2022 I decided to try to identify the strings for all of th entries in Mojang's server blocklist. Through many different methods and approaches, including:
+In September of 2022 I decided to try to identify the strings for all of the entries in Mojang's server blocklist. Through many different methods and approaches, including:
 
 - bruteforcing
 - pulling domains from server lists
@@ -14,6 +14,14 @@ I was able to identify many new strings in the list.
 
 There's some GitHub automation in place to automatically update everything every couple of hours.
 
+## How to help
+
+Obviously, we want to continue to identify more strings. That's really it. If you are able to identify more, please open a PR or an issue!
+
+Take a look at data/todo.txt for hashes that have yet to be cracked.
+
+If you run a Minecraft server list site, you are exactly the kind of person we're looking for that has the resources to help!
+
 ## How to use this stuff
 
 - data/current.txt contains the current blocklist, as fetched from https://sessionserver.mojang.com/blockedservers
@@ -22,7 +30,9 @@ There's some GitHub automation in place to automatically update everything every
 
 ## Adding new stuff
 
-Throw whatever you want at `node try_url.js`. See scratchwork.md for various examples of usage. If you find something new, run this stuff:
+See scratchwork.md for various neat and useful snippets.
+
+Throw whatever you want at `node try_url.js`. If you find something new, run this stuff:
 
 ```
 npm run update-blocklist ; npm run update-merged; npm run update-todo
@@ -32,26 +42,25 @@ For some reason, `update-todo` sometimes fails on certain systems, no clue why,
 
 Don't prune identified strings that have been removed from identified.txt -- I'm keeping them in there for historical purposes. I might end up adding a separate file for removed strings at some point which would include verified former blocklist entries.
 
+Take a look at the `expandmc` bash function in scratchwork.md
+
 ## Background information on the blocklist
 
-For a server to reach some level of popularity and discoverability, it must either have 
+Beginning at Minecraft 1.9.3 r2, Mojang started blocking certain Minecraft servers using a blocklist.
-a persistent hostname, or have its IP known. 
 
-Large lists of Minecraft servers are generally unpublished, to reduce risk
+From [wiki.vg's documentation](https://wiki.vg/Mojang_API#Blocked_Servers):
-of random 'griefing' (DoSing or otherwise harassing servers). Perhaps these
-servers can still be used in other ways, even when blocked by Mojang.
-Mojang apparently started using this blocklist method on May 1, 2016.
 
-There appear to be three classes of entries:
+> _Clients check the lowercase name, using the ISO-8859-1 charset, against this list. They will also attempt to check subdomains, replacing each level with a `*`. Specifically, it splits based off of the `.` in the domain, goes through each section removing one at a time. For instance, for mc.example.com, it would try `mc.example.com`, `*.example.com`, and `*.com`. With IP addresses (verified by having 4 split sections, with each section being a valid integer between 0 and 255, inclusive) substitution starts from the end, so for `192.168.0.1`, it would try `192.168.0.1`, `192.168.0.*`, `192.168.*`, and `192.*`._
+
+> _This check is done by the bootstrap class in netty. The default netty class is overridden by one in the `com.mojang:netty` dependency loaded by the launcher. This allows it to affect any version that used netty (1.7+)_
+
+The blocklist appears to have three classes of entries:
 
 ### Hostnames
 
-This including wildcards, typos, '?' appended, mixed case, and other anomalies.
+This includes wildcards, typos, '?' appended, mixed case, and other anomalies. Minecraft servers don't need to know their own hostname to function, so scanning the Internet for servers isn't very useful (except maybe with rDNS). Checking server lists is a decent way to find them though.
-Minecraft servers don't need to know their own hostname to 
+
-function, so many simple setups have no detectable distinct hostname at the
+There used to be a way to bypass blocks via rotating SRV records, but this was patched in snapshot 21w13a (Mojira issue MC-136551, "Servers able to bypass EULA blacklist.")
-Minecraft level - so scanning the Internet for servers is not very useful.
-But looking for lists of servers that allow cheating, or known to have weak
-anti-cheating measures, is effective.
 
 ### IPs
 
@@ -60,16 +69,16 @@ This set of hashcat masks for all valid IP addresses can be run after every new
 
 https://github.com/johnjohnsp1/hexhosts/blob/master/ipv4.hcmask
 
+There aren't many of these. Mojang does not block many IPs in the list. From some of our research, we found that they _used to_ but after one notable incident where they blocked ProxyPipe, they generally shifted direction towards blocking hostnames.
+
 ### Test entries
 
-These are not valid DNS FQDNs, or even hostnames (some have spaces, underscores, etc)
+These are not valid DNS FQDNs, or even hostnames (some have spaces, underscores, etc). These often have "dns" and/or "test" in them, with various combinations of separators (including space), case, and appended digits.
-These often have "dns" and/or "test" in them, with various combinations of 
-separators (including space), case, and appended digits.
 
 ## Thanks
 
 Special thanks to:
 
-- @roycewilliams - who has provided a lot of help with identifying various strings.
+- @roycewilliams - who has provided a lot of help with identifying various strings through research.
-- All the people who have put in work to identify hashes in the past
+- All the people who have put in work to identify hashes in the past (e.g @Reecepbcups, @theminecoder)
 - Various people who have let me look at their data (even if it didn't identify (m)any new hashes) like @Yive and some server list owners

scratchwork.md

@@ -1,6 +1,31 @@
 # Notes/scratchwork 
 
-## Useful bash snippets
+## bash functions
+
+```
+# expand string to common patterns (feel free to add more subdomains, like "hub", "jugar", "server", etc)
+expandmc () { 
+    awk '{print $1" play."$1" mc."$1" _minecraft._tcp."$1}' | tr ' ' '\n' | awk '{print $1" *."$1}' | tr ' ' '\n'
+}
+# space to newline
+s2n () {
+  tr ' ' '\n'
+}
+```
+
+```sh
+$ echo example.com | expandmc 
+example.com
+*.example.com
+play.example.com
+*.play.example.com
+mc.example.com
+*.mc.example.com
+_minecraft._tcp.example.com
+*._minecraft._tcp.example.com
+```
+
+## Various one-liners and bash snippets
 
 Get a list of TLDs (idk if this is super up to date)
 
@@ -8,22 +33,15 @@ Get a list of TLDs (idk if this is super up to date)
 curl -s https://raw.githubusercontent.com/umpirsky/tld-list/master/data/en/tld.txt|grep -Po "\(\K.+?(?=\))" > tld.txt
 ```
 
-Expand from mc-server-list-scraper
-
 To strip the first subdomain (will make the other subdomains more likely to work), throw this in the mix: `grep -Po "\.\K.*"`
 
-```
+Get the middle segment (part before the TLD) of all entries, excluding ddns.net, spit it out as `*.string`
-awk -F/ '{print $NF}' ../mc-server-list-scraper/results/* |awk -F: '{print $1}' | awk '{print $1" *."$1" *.play."$1" *.mc."$1" play."$1" mc."$1" hub."$1" *.hub."$1" *.minecraft."$1" minecraft."$1" *.jugar."$1" jugar."$1}'|s2n| sort -u | pv -l | xargs -P2 node try_url.js
-```
-
-
-Get the middle segment (part before the TLD) of all entries, excluding ddns.net, spit it out as *.string
 
 ```
 awk -F= '{print $2}' data/identified.txt|grep -v ddns|awk -F. '{print $(NF-1)}'|sort -u > middle_segments.txt
 ```
 
-For all TLDs in tld.txt, try *.string.tld (try also: no subdomain, `play.`, `mc.`, etc)
+For all TLDs in tld.txt, try `*.string.tld` (try also: no subdomain, `play.`, `mc.`, etc)
 
 ```
 for tld in $(cat tld.txt); do cat middle_segments.txt|awk '{print $1".'$tld'"}';  done|pv -l |xargs -P3 node try_url.js
@@ -37,13 +55,13 @@ comm -23 <(sort -u data/current.txt) <(awk -F= '{print $1}' data/identified.txt
 
 ### for big lists of minecraft server urls:
 
-remove first subdomain. replace with *.<domain>. this also strips port numbers and normalizes casing
+remove first subdomain. replace with `*.<domain>`. this also strips port numbers and normalizes casing
 
 ```
 cat minecraftservers_org_scrape.txt| grep -Po ".+?(?=:)" | grep -Po ".+?(?=\.)\K.*" | tr '[[:upper:]]' '[[:lower:]]'|awk '{print "*"$1}'|xargs node try_url.js
 ```
 
-Do srv lookups for a list of domains
+### Do srv lookups for a list of domains
 
 ```
 cat domains.txt| grep -Po ".+?(?=:)" | tr '[[:upper:]]' '[[:lower:]]'|grep [[:alpha:]]| xargs -I{} -P10 timeout 5 dig srv _minecraft._tcp.{} +short | tee -a domains_srv_resolved.txt 
@@ -61,7 +79,7 @@ another thingy similar
 cat minecraftservers_org_scrape_resolved_srv.txt | tr ' ' '\n'|egrep [[:alpha:]]|grep -Po ".+?(?=\.$)"|tr [:upper:] [:lower:]| sort -u| awk '{print $1" *."$1" play."$1}' | xargs node try_url.js
 ```
 
-try *.mc or *.play subdomains for existing
+try `*.mc` or `*.play` subdomains for existing
 
 ```
 awk -F= '{print $NF}' data/identified.txt |grep [[:alpha:]]|grep -Po "\*\.\K.*"|awk '{print "*.mc."$1}'|xargs node try_url.js
@@ -69,7 +87,6 @@ awk -F= '{print $NF}' data/identified.txt |grep [[:alpha:]]|grep -Po "\*\.\K.*"|
 
 Finding bypassers via SRV...
 
-
 ```
 awk -F= '{print $2}' data/identified.txt |sed 's/*.//'|awk '{print "_minecraft._tcp."$1}'|xargs -L1 -P10 dig +short srv |tee srv_re_resolve.txt
 cat srv_re_resolve.txt |awk '{print $NF}'|sed 's/\.$//'|xargs node try_url.js