Human user or Java robots?
How to trick websites into thinking that your program is human.

In Spain, the central register that I mentioned earlier is called the "Catastro". It contains a rich description of every property on the territory. It can be publicly accessed at this address. 

Upon accessing the website, you will be faced with a field where you have to input the property identifier (the Referencia catastral). For this example, we will use the Referencia catastral 4675602DG8747N0028HL. This number can be found, for instance, when you rent a flat or when you buy a property. It can also be found on real estate searche engines like Idealista when that property is on the market. There is some structure in that number: the first seven digits identify the property or plot, the next seven indicate the sheet of the plan where it is located, the next four identify the property within the property and, finally, the last two digits are control characters.

I illustrate the steps using MATLAB but there exist other softwares with the same capabilities.

Download my programs

• The Java robot and human emulation code

• The robotType function

Walk-through of the program

Build a typical request

Before running large loops to gather your data, it is always best to trial it with a few items first. 

The first step is to run it manually once and identify with part of the URL you will target changes with each request. The typical URL looks like this:

https://www1.sedecatastro.gob.es/CYCBienInmueble/OVCConCiud.aspx?UrbRus=U&RefC=4675602DG8747N0028HL&esBice=&RCBice1=&RCBice2=&DenoBice=&from=OVCBusqueda&pest=rc&RCCompleta=4675602DG8747N0028HL&final=&del=17&mun=95.

Let's break it down in bits:

• https://www1.sedecatastro.gob.es/CYCBienInmueble/OVCConCiud.aspx?UrbRus=U  : this part is the beginning of the URL and will not change from one request to another. This is what I call the root URL.

• &RefC=4675602DG8747N0028HL : this part is the field where we will input the Referencia Catastral. The part in red will change for every request.

• &esBice=&RCBice1=&RCBice2=&DenoBice=&from=OVCBusqueda&pest=rc&RCCompleta=4675602DG8747N0028HL&final= : those fields are optional and will not change across requests.

• &del=17&mun=95 : those bits represent the province and municipality, respectively. Those codes can be found at the National Statistical Office.

All in all, our destination URL will look like this: root URL + Referencia catastral + optional fields + Province code + Municipality code. Only the red elements will change from one request to another.

Invoke the robot and define the emulation environment

To be able to send keystrokes and mouse clicks, one needs to start two things:

1. robot = java.awt.Robot; % Invoke the robot.

2. h = actxserver('WScript.Shell'); % Start the emulation environment.

The first part loads the Java environment while the second creates the link between MATLAB and shell commands.

We will use four functions of the Java robot: 

• keyPress/keyRelease sends the instructions of pressing and releasing keyboard keys

• mousePress/mouseRelease sends the instruction of clicking and releasing a mouse button

Similarly, we can send keystrokes within the Shell environment with the SendKeys function. 

A very quick guide to robots in MATLAB

The robot and shell environments can be combined to generate more elaborate instructions. For instance, an instruction for a CTRL+SHIFT+S (Save as...) will be translated as:

eval(['robot.keyPress(java.awt.event.KeyEvent.VK_CONTROL)']);

eval(['robot.keyPress(java.awt.event.KeyEvent.VK_SHIFT)']);

h.SendKeys(['s']);

eval(['robot.keyRelease(java.awt.event.KeyEvent.VK_SHIFT)']);

eval(['robot.keyRelease(java.awt.event.KeyEvent.VK_CONTROL)']);

In terms of mouse actions, let us consider a simple left-click:

robot.mousePress(java.awt.event.InputEvent.BUTTON1_MASK);

robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

For a double-click, simply repeat the instructions above.

Lastly, we need to  wrap the SendKeys tool into a function so that we can emulate a human user by executing the function robotType adapted from this thread as: 

robotType('Hello World.');

The robotType function will check, character by character, if the character is numerical, a string, a white space, or some punctuation. Accordingly, it will check if the character needs to be inputed as an upper or lowercase letter. Special characters are also dealt with using a combination of ALT+Numbers. robotType will then input the character accordingly.

You can find more information about Java robots in MATLAB here . The list of class elements available to the robot can be found here .

Define the steps of your algorithm

Next, we need to record every single step in the algorithm that the Java robot needs to follow and transpose it into a language it will understand. We need to be exhaustive and break every step into the most basic bit.

Open Chrome (or any other browser)

1. h.Run('chrome'); %Invokes chrome.exe

2. pause(3); %Waits for the application to load.

3. h.AppActivate('Chrome'); %Brings Chrome to focus

Input the URL

1. Maximize the window (ALT+SPACE+X):

eval(['robot.keyPress(java.awt.event.KeyEvent.VK_ALT)'])

eval(['robot.keyPress(java.awt.event.KeyEvent.VK_SPACE)'])

h.SendKeys(['x']);

eval(['robot.keyRelease(java.awt.event.KeyEvent.VK_SPACE)'])

eval(['robot.keyRelease(java.awt.event.KeyEvent.VK_ALT)'])

2. Move the pointer to the URL bar:

x0=200; % X-coordinate of the pointer. Screen-dependent

y0=820; % Y-oordinate of the pointer. Screen-dependent

set(0,'PointerLocation',[x0 y0]); % Move the pointer to the given coordinates

3. Highlight the URL bar and input the destination URL, ending with the ENTER key:

robot.mousePress(java.awt.event.InputEvent.BUTTON1_MASK);

robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

h.SendKeys([destinationURL '~']); % URL of the referencia catastral

Obtain the data

For this part of the walk-through, I will only list the steps and not the code that goes with it since these steps are essentially an application of what I have already described above. You can see the steps in the video above.

1. Input the property ID in the field

2. Move the pointer to the button "DATOS"

3. Click the button

Save the page

1. Once the page has loaded, right-click

2. Move to the Save as... option and left-click it

3. Once the Save window has opened, move to the folder field, remove the contents and input the folder path:

filepath='C:\Catastro\';

robotType(filepath);

4. Move to the file field, remove the contents and input the name of the file:

filename=strcat('Ref_',Ref_catastral.ref_catastral{a,1},'.html'); % Ref_catastral is a structure where the catastral reference, the province number and the municipiality number are stored

robotType(filename);

5. Move to the Save button and click it

6. Confirm if file is to be overwritten (move and click)

7. Close the browser with ALT+F4:

h.SendKeys('%{F4}'); % close Alt+F4

Once you have reached the end of your first iteration, loop over all the other property IDs that you have to check, and voilà!

Process and save the data

Once we have saved the HTML file, we will need to filter it to extract the information we need.

1. Open the HTML file as text. This file will contain the source code of the page we just downloaded.

% Open the saved HTML document

fileID = fopen(strcat(cd,'\HTML\','Ref_',Ref_catastral.ref_catastral{a,1},'.html'),'r');

outURL = fscanf(fileID,'%c');

close(fileID); 

where a is the iteration in your loop.

2. For each field that we are interested in, we will have to identify how it appears in the HTML code and isolate that.

localizacionstr='<span class="col-md-4 control-label ">Localización</span><div class="col-md-8 "><span class="control-label black"><label class="control-label black text-left">';

lengthLOC=strlength(localizacionstr);

StrfindLOC=strfind(outURL,localizacionstr);

The steps are often the same: (i) define the string you are looking for, (ii) compute the length of the string, (iii) find it in the string of characters, (iv) select that string from the first character found to the length of the string you are trying to find.

3. [Optional] Some cleaning may be required with tools that apply to strings (strtrim, strrep, etc) and depend on the complexitiy of the information you want to extract. Given the lack of harmonization in the type of properties (condos, semi-detached, etc), some fields will be relevant for that iteration and others not.

UsefulString=outURL(StrfindLOC+lengthLOC:StrfindCLASE-1); % Reduce the size of where we have to look

UsefulString=strrep(UsefulString,'Pl: ','Pl:'); % Harmonize the information

UsefulString=strrep(UsefulString,'Pt: ','Pt:');

UsefulString=strrep(UsefulString,'Es: ','Es:');

UsefulString=strrep(UsefulString,'Bl: ','Bl:');

UsefulString=strrep(UsefulString,'0000','');

UsefulString=strrep(UsefulString,'---','');

Lines=strsplit(UsefulString,'<br>'); % Find line breaks

Lines=strtrim(Lines);

StrfindPL=strfind(Lines{1},'Pl:'); % Find the precise location of the property

StrfindPT=strfind(Lines{1},'Pt:');

StrfindES=strfind(Lines{1},'Es:');

StrfindBL=strfind(Lines{1},'Bl:');

Planta=UsefulString(StrfindPL+3:StrfindPL+4);

Puerta=UsefulString(StrfindPT+3:StrfindPT+4);

Escalera=UsefulString(StrfindES+3:StrfindES+4);

Bloque=UsefulString(StrfindBL+3:StrfindBL+4);

temp=strsplit(Lines{1},' ');

StreetType=temp{1};

if numel(temp)==2

    StreetNumber=temp{end- ~isempty(Planta) - ~isempty(Puerta) - ~isempty(Escalera) - ~isempty(Bloque)-1};

    StreetName=strjoin(temp(2:end-~isempty(Planta) - ~isempty(Puerta) - ~isempty(Escalera) - ~isempty(Bloque)),' ');

else

    StreetNumber=temp{end- ~isempty(Planta) - ~isempty(Puerta) - ~isempty(Escalera) - ~isempty(Bloque)};

    StreetName=strjoin(temp(2:end-~isempty(Planta) - ~isempty(Puerta) - ~isempty(Escalera) - ~isempty(Bloque)-1),' ');

end

temp2=strsplit(Lines{2},' ');

temp3=strsplit(Lines{2},'(');

PostalCode=temp2{1};

ProvinceName=temp3{end}(1:end-1);

MunicipalityName=strjoin(temp2(2:end-~isempty(ProvinceName)),' ');

StrfindOpenBracket=strfind(MunicipalityName,'(');

if ~isempty(StrfindOpenBracket)

    MunicipalityName=MunicipalityName(1:StrfindOpenBracket-2);

else

end

4. Populate the table

tabletemp=[{Ref_catastral.ref_catastral{a,1}},{ProvinceName},{MunicipalityName},{PostalCode},{StreetType},{StreetName},{StreetNumber},{Escalera},{Planta},{Puerta},{SuperficieVivienda},{SuperficieOtroConstruido},{RenovationType},{RenovationDate},{Class},{TypeUse},{YearBuilt},{PlotType},{PlotSize},{PlotShare},{a}];

Table{a,:}=tabletemp ;

5. Save

save ScrapingCatastro Table a;

Tips and Tricks

• Don't be greedy! This emulation is a second-best solution for when websave does not work. Some website may limit the number of requests an IP address can make in a given time. Try to stay below that limit.

• Don't hesitate to add pauses to your program for the actions that can require more time. Always be on the safe side and provide some headroom.

• Experiment with what works best for you. Coding is highly personal, with habits developed along the years. If there are parts that don't satisfy you, don't hesitate to change them.

Final considerations

The code may need to be adapted in several aspects. First, you may be interested in another application than the one I just described. 

Second, one way to improve the code could be to dynamically adapt the X- and Y-coordinates of the pointer according to the current screen size rather than having a fixed location. This would prove to be more reliable if the code is run on another computer. Adapting the first iteration would look like this:

sz=get(0,'ScreenSize');

x0=0.1302*sz(3);

y0=0.9491*sz(4);

Download my program

• The complete program

What is a reCAPTCHA?

A reCAPTCHA is a free service offered by Google to protect websites from spam and abuse. It acts as a “CAPTCHA” (Completely Automated Public Turing test to tell Computers and Humans Apart), which serves to distinguish between humans and bots. While humans can easily solve CAPTCHAs, they pose a challenge for automated softwares and malicious bots.

 In reality, it is not really clicking the box itself that triggers the validation. In fact, Google evaluates your behavior both before and after clicking the checkbox to determine whether you appear human. This assessment includes various factors, such as your browsing history or the movement of your mouse.

What do you mean by 'solving a reCAPTCHA'?

Solving a reCAPTCHA is essentially making your computer program behave in a sufficiently human-like fashion. To do so, it needs to match the speed of movement of a human user and the randomness in her behavior. In short, it has to be perfectly imperfect.

Walk-through of the algorithm

As a first step, load the page that has a reCAPTCHA using the robot I mentioned in the first section. Then, follow the steps below:

1. Invoke the robot and load the page:

destinationURL='https://www.google.com/recaptcha/api2/demo?invisible=false';

h = actxserver('WScript.Shell'); % Invoke the command window

h.AppActivate('Chrome'); % Bring Chrome to focus

h.SendKeys([URL '~']); % Sends keystrokes

robot = java.awt.Robot(); % Invoke the robot

2. Take a screenshot

pos = [0 0 sz(3) sz(4)]; % [left top width height]

rect = java.awt.Rectangle(pos(1),pos(2),pos(3),pos(4));

cap = robot.createScreenCapture(rect);

3. Transform the screenshot so that MATLAB can use it

% Convert to an RGB image

rgb = typecast(cap.getRGB(0,0,cap.getWidth,cap.getHeight,[],0,cap.getWidth),'uint8');

imgData = zeros(cap.getHeight,cap.getWidth,3,'uint8');

imgData(:,:,1) = reshape(rgb(3:4:end),cap.getWidth,[])';

imgData(:,:,2) = reshape(rgb(2:4:end),cap.getWidth,[])';

imgData(:,:,3) = reshape(rgb(1:4:end),cap.getWidth,[])';

% Show or save to file

figure;imshow(imgData);

imwrite(imgData,'out.png'); 

The relevant square (the square we have to click) is a black square surrounded by a white rectangle.

img=imread('out.png');

Ibw=~im2bw(img,0.999); % transform to black and white

Iwb=~Ibw; % Invert the black and white

4. Find the relevant square

% Find the black pixels

[rfind,cfind]=ind2sub([size(Ibw)],find(Ibw==0));

unique_rows=unique(rfind);

unique_cols=unique(cfind);

Going row by row, identify the black pixels and the length of continuous pixels of the same color.

PotentialBlackSquare=cell(sz(4),1);

WtoB=cell(sz(4),1);

BtoW=cell(sz(4),1);

for i=1:sz(4) % innermost black line

    try

        contiguouswhite=[0 diff(Ibw(unique_rows(i),:))];

        WtoB{i,1}=find(contiguouswhite==-1);

        BtoW{i,1}=find(contiguouswhite==1);

        switches=[sort([1 unique(BtoW{i,1}) unique(WtoB{i,1}) sz(3)])];

        iswhite=repmat([0 1],1,10);

        iswhite=iswhite(1:length(switches)); % merge the black to white or white to black based on which one is first

        Lengthsegments=diff(switches);

        notcandidate=Lengthsegments>50 | Lengthsegments<5; % too big or too small

        candidate= find(notcandidate==0 & iswhite(2:end)==0)+1; % get the reference line in switches

        PotentialBlackSquare{i,1}=[switches(candidate) switches(candidate+1)];

    end

end

not2=cellfun(@(x) size(x,2),PotentialBlackSquare,'UniformOutput',true);

PotentialBlackSquare(not2~=2,1)={[0 0]};

notrightsize=cellfun(@(x) x(2)-x(1),PotentialBlackSquare,'UniformOutput',true);

PotentialBlackSquare(notrightsize<5 | notrightsize>50,1)={[0 0]};

notrightsize=cellfun(@(x) x(2)-x(1),PotentialBlackSquare,'UniformOutput',true);

5. Since we are looking for a square, the number of rows after the candidate should be the same as the number of pixels in that row.

findcandidate=find(notrightsize~=0);

notrightsize2=notrightsize;

for i=1:numel(findcandidate)

    tester = all([notrightsize(findcandidate(i):findcandidate(i)+notrightsize(findcandidate(i))-1)]==notrightsize(findcandidate(i)));

    if tester==0

        notrightsize(findcandidate(i))=0;

    end

end

topleft.Y=find(notrightsize~=0);

topleft.X=PotentialBlackSquare{topleft.Y,1};

6. Define the center of the square we need to click inside of.

centersquare.X=topleft.X(1)+ floor(0.5*(topleft.X(2)-topleft.X(1)));

centersquare.Y=topleft.Y(1) + floor(0.5*(topleft.X(2)-topleft.X(1))); % coordinates from the top left corner of the screen

7. Move the pointer from the current location to the destination location

% Click inside the square

h.AppActivate('Chrome');

y0=sz(4)-centersquare.Y; % invert the ordering of the Y-axis

currentlocation=get(0,'PointerLocation'); % Get current cursor location

difflocation.x=x0-currentlocation(1); % Compute difference of coordinates

difflocation.y=y0-currentlocation(2);

pathlocation.x=linspace(currentlocation(1),x0,30); % Generate path

pathlocation.y=linspace(currentlocation(2),y0,30);

pathlocation.x=pathlocation.x+2*randn(size(pathlocation.x)); % Add randomness to the path

pathlocation.y=pathlocation.y+2*randn(size(pathlocation.y));

pathlocation.x(end+1)=x0; % Define final location

pathlocation.y(end+1)=y0;

for i=1:numel(pathlocation.x)

    set(0,'PointerLocation',[pathlocation.x(i) pathlocation.y(i)]); % Move along the path

    pause(0.1+1/100*rand());

end

8. Left-click

set(0,'PointerLocation',[x0 y0]);

pause(0.2+1/10*rand());

robot.mousePress(java.awt.event.InputEvent.BUTTON1_MASK); % Press on the square

pause(0.01+1/100*rand());

robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

pause(0.01);

9. Check that the click was successful

10. Click on the button

x1=x0+randn();

y1=y0-0.01*i*sz(4)-randn();

currentlocation=get(0,'PointerLocation'); % Get current cursor location

difflocation.x=x1-currentlocation(1); % Compute difference of coordinates

difflocation.y=y1-currentlocation(2);

pathlocation.x=linspace(currentlocation(1),x1,10); % Generate path

pathlocation.y=linspace(currentlocation(2),y1,10);

pathlocation.x=pathlocation.x+2*randn(size(pathlocation.x)); % Add randomness to the path

pathlocation.y=pathlocation.y+2*randn(size(pathlocation.y));

pathlocation.x(end+1)=x1; % Define final location

pathlocation.y(end+1)=y1;

for i=1:numel(pathlocation.x)

    set(0,'PointerLocation',[pathlocation.x(i) pathlocation.y(i)]); % Move along the path

    pause(0.1);

end

set(0,'PointerLocation',[x1 y1]);

pause(0.2+1/10*rand());

robot.mousePress(java.awt.event.InputEvent.BUTTON1_MASK); % Press on the button

pause(0.01+1/100*rand());

robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

Further considerations

There are some limitations to the program I have explained above. First, it only works if we are faced with a checkbox reCAPTCHA.  If we have to click on images containing firetrucks or school buses, this program cannot be used. Second, I have determined that the object we were looking for was a square surrounded by a white rectangle. If the website decides to randomly change colors for its CAPTCHA, this code may not be able to beat the CAPTCHA. Similarly, if the object changes from a square to a rectangle randomly, this code may not be useful.

Nevertheless, these limitations can be overcome with some tweaks in the code.

To truly generate human behavior, we could also generate a fake browsing history. What do you do in the morning? You may want to check the weather, then go on to your favorite news site, or check your emails, etc. Adding mouse movements and scrolling on those pages may help build credibility as a human user, at a very small cost in terms of time before you even launch your automated program.

Finally, the movement of the mouse could be wrapped into a function order to reduce the number of lines of code. This function could take as argument the destination of the pointer, whether we want a simple movement or a click, whether we want it quick or slow, and whether we want to move the mouse with a certain degree of randomness or not.