Field Study

1.) Number of CAPTCHA-Cloaked Phishing Websites

We conducted the experiment for about 3 weeks and crawled about 500,000 zero-day websites, The system detected 1024 websites using CAPTCHA, of which 175 websites are reported as CAPTCHA-cloaked phishing websites. The tables below show various statistics related to experiment.

Table 1: Confusion Matrix for CAPTCHA detection.

First, we analyze the performance of PhishDecloaker's detection and recognition models on webpages in-the-wild. Through inspecting all webpages with a positive CAPTCHA prediction, we can determine true and false positives. On the other hand, true and false negatives are estimated by sampling and inspecting a subset of webpages with negative CAPTCHA prediction. From Table 1, the precision and recall are calculated as 0.85 and 0.92, respectively. 

Misclassification of CAPTCHAs Resembling Non-CAPTCHA Elements

The main factor affecting PhishDecloaker's performance is the misclassification of CAPTCHAs resembling non-CAPTCHA elments. Non-CAPTCHA elements (e.g., logos, buttons) were erroneously detected as text-based and press & hold CAPTCHAs, whereas new text-based CAPTCHA variants went undetected.  To reduce false positives on CAPTCHAs, we suggest incorporating other heuristics to further confirm their presence on the page. For example, by checking for input fields in the vicinity of the detection. We noticed that text-based CAPTCHA are commonly used by non-English-language websites, and press & hold CAPTCHAs can be traced to commercial anti-bot protection services (i.e., HUMAN, PerimeterX). Our analysis did not reveal any instances of phishing websites using the aforementioned CAPTCHA types.

Out-of-distribution CAPTCHA Challenges

hCaptcha periodically releases new challenge types that are conceptually different from its existing challenges. While it is possible to actively train deep learning models against each new challenge type, it is inefficient as some challenge types are temporary (i.e., for A/B testing and feature rollout). Currently, we instruct our solver to skip the new challenges and request new ones until a compatible challenge is found.

Custom CAPTCHA Variants

Phishdecloaker's performance is affected by custom CAPTCHA variants. While the system can identify and categorize these CAPTCHAs, its could not solve them due to the absence of corresponding solvers or the CAPTCHA's incompatibility with existing solvers. For instance, we've encountered cases where phishers replicate the form and functionality of reCAPTCHAv2, designed to thwart signature based detections. Additionally, we've identified a case involving slider-based CAPTCHAs, with sliders modified to move diagonally instead of horizontally. Solving this CAPTCHA redirects us to a phishing page posing as Bet365. To effectively address custom CAPTCHAs, we propose two potential directions: 1) Develop a more robust model for out-of-distribution (OOD) CAPTCHA detection, maintain a set of common CAPTCHA solvers, and use human operators as a fallback for solving unknown CAPTCHAs, as in PhishDecloaker's case. 2) Strive towards the development of generalist agents for web crawling tasks.

Multiple Anti-Bot Protection Layers

Websites, whether phishing or benign, often have robust anti-bot protection, These protective measures encompass various aspects, including network information (i.e., IP blacklist, geo-blocking, TLS handshake), browser identity (i.e., user agents, request headers, canvas fingerprints, audio fingerprints, WebGL rendering, media devices), and behavioral data (i.e., visit frequency, mouse movements, keyboard presses).  In such cases, despite answering the CAPTCHA challenge, our crawler will still be rejected for being suspicious. We believe that by partnering with anti-bot companies (i.e., Cloudflare's friendly-bot allowlist), and employing strategic workarounds (i.e., residential proxies), we can minimize the impact of other human authentication factors.