Squid
------
---------
MIKROTIK :Howto Redirect HTTP traffic to SQUID with Original Source Client IP
Filed under: Linux Related, Mikrotik Related — Tags: mikrotik howto route http traffic to squid, mikrotik routing http to squid, Mikrotik with Squid, redirect users with original source ip address, Squid ZPH, transparent Squid with ZPH — Syed Jahanzaib / Pinochio~:) @ 9:22 AM
35 Votes
Howto connect SQUID box with Mikrotik and Log user’s original source ip in squid proxy log.
Consider the following Scenario.
Mikrotik + SQUID + Client Source IP Loggin
In this scenario Mikrotik is acting as PPPoE Server. All internet traffic is passed from Mikrotik, except the HTTP PORT 80 data, which is marked and route to SQUID proxy server for caching facility and some other filtering task. We want to preserve source client IP address in squid logs, so that user web activity can be tracked via squid access log, sometimes it can be require by law enforcing agencies or can be required by administration / reporting / management purposes. So how to do it ?
I assume you already have working SQUID and Mikrotik (pppoe server with pppoe user ip pool 172.16.0.0/16 pool in place. however you can change it according to your scenario)
We will divide this article in two sections
1# MIKROTIK RouterOS Configuration
2# SQUID Proxy Server Configuration
1# MIKROTIK RouterOS CONFIGURATION !
In this example, Mikrotik have 3 LAN interfaces, Details are as following . . .
1
2
3
4
5
[admin@MikroTik] > ip address print
# ADDRESS NETWORK INTERFACE
0 192.168.0.1/24 192.168.0.0 ether1 >> LAN INTERFACE
1 192.168.2.1/24 192.168.2.0 ether2 >> PROXY INTERFACE
2 192.168.1.2/24 192.168.1.0 ether3 >> WAN - DSL INTERFACE
Now we will start with the Mikrotik configuration:
# Add ip addresses for interface cards,
1
2
3
4
1
1
2
3
4
5
# Set DNS for Host Resolving
/ip address
add address=192.168.0.1/24 disabled=no interface=ether1 network=192.168.0.0
add address=192.168.2.1/24 disabled=no interface=ether2 network=192.168.2.0
add address=192.168.1.2/24 disabled=no interface=ether3 network=192.168.1.0
/ip dns set allow-remote-requests=yes cache-max-ttl=1w cache-size=8000KiB max-udp-packet-size=512 servers=192.168.1.1
# Mark All HTTP Port 80 Traffic, so that we can use these Marked Packets in Route section.
/ip firewall nat
add action=accept chain=srcnat disabled=no dst-port=80 protocol=tcp
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=no dst-port=80 new-routing-mark=http passthrough=yes protocol=tcp
# Masquerade all traffic (Except http] on ether3 Only, which is connected with DSL Router. This is important to masquerade traffic on WAN Interface only, otherwise http packets will also be masqueraded with mikrotik ip.
1
/ip firewall nat add action=masquerade chain=srcnat disabled=no out-interface=ether3
# Define Route for HTTP Marked packets, and set default rule for all other traffic, This is called policy base or pre traffic base routing
1
2
3
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.2.2 routing-mark=http scope=30 target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.1 scope=30 target-scope=10
(Where 192.168.2.2 is the Squid Proxy Server LAN IP , and 192.168.1.1 is our DSL Router IP)
That’s all for MIKROTIK, Now Mikrotik will Redirect HTTP Traffic to Squid Proxy via interface ether2. and all rest of traffic will be masqueraded/nat to WAN (ether3) which is connected with DSL.
Now moving on to SQUID section !
2# SQUID Proxy Server CONFIGURATION !
I assume you already have working squid in transparent mode [its necessary you set this via iptables and some directives in squid.conf
For example: (squid.conf)
1
2
3
4
5
6
1
2
and IPTABLES example:
#==============================
#Transparent Mode & Example ACL
#==============================
http_port 8080 transparent
acl mylocalnet src 0.0.0.0/0.0.0.0
http_access allow mylocalnet
# DNAT port 80 request comming from LAN systems to squid 8080 aka transparent proxy .zaib.
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to 192.168.2.2:8080
In this example, Squid proxy server have 2 Interface cards
eth0: LAN (connected with Mikrotik’s PROXY INTERFACE ether2) = 192.168.2.2
eth1: WAN (connected with DSL Router) = 192.168.1.3 / Default GW = 192.168.1.1 / DNS = 192.168.1.1 , 8.8.8.8
It is necessary that SQUID can directly communicate with the users by setting ROUTE to communicate user subnet via mikrotik, otherwise it won’t be able to communicate with the user , Issue the following command ,
1
route add -net 172.16.0.0 netmask 255.255.0.0 gw 192.168.2.1 dev eth0
172.16.0.0/16 is pppoe user IP pool, After successfully execution, Squid will be able to see the Users ip. Also add the above route command in /etc/rc.local (u must add any command in rc.local before ‘exit 0’ if using ubuntu)
Above command Explanation: 172.16.0.0 are pppoe users ip pool and 192.168.2.1 is Mikrotik ether2 which is directly connected to Squid via crossover cable , thus we are telling Squid to look after for users 172.16.0.0 via gateway 192.168.2.1 which is mikrotik, if we don’t use this, squid and users wont be able to communicate with each other)
That’s it. Now when user will try to use internet, his HTTP traffic will be marked by Mikrotik and then all HTTP marked traffic will be ROUTED to SQUID proxy [192.168.2.2] with original client ip, instead of Mikrotik ip.
Some useful links for fine tuned working squid.conf and Linux internet sharing script.
Youtube caching with Squid + Nginx
Filed under: Linux Related — Tags: nginx cache, youtube cache, youtube cache with nginx — Syed Jahanzaib / Pinochio~:) @ 11:52 AM
41 Votes
Updated Version of squid which cache youtube and many other contents. read following
https://aacable.wordpress.com/2012/01/19/youtube-caching-with-squid-2-7-using-storeurl-pl/
Advantages of Youtube Caching !!!
In most part of the world, bandwidth is very expensive, therefore it is (in some scenarios) very useful to Cache Youtube videos or any other flash videos, so if one of user downloads video / flash file , why again the same user or other user can’t download the same file from the CACHE, why he sucking the internet pipe for same content again n again?
Peoples on same LAN ,sometimes watch similar videos. If I put some youtube video link on on FACEBOOK, TWITTER or likewise , and all my friend will watch that video and that particular video gets viewed many times in few hours. Usually the videos are shared over facebook or other social networking sites so the chances are high for multiple hits per popular videos for my LAN users / friends. [syed.jahanzaib]
This is the reason why I wrote this article.
Disadvantages of Youtube Caching !!!
The chances, that another user will watch the same video, is really slim. if I search for something specific on youtube, i get more then hundreds of search results for same video. What is the chance that another user will search for the same thing, and will click on the same link / result? Youtube hosts more than 10 million videos. Which is too much to cache anyway. You need lot of space to cache videos. Also accordingly you will be needing ultra modern fast hardware with tons of SPACE to handle such kind of cache giant. anyhow Try it
AFAIK you are not supposed to cache youtube videos, youtube don’t like it. I don’t understand why. Probably because their ranking mechanism relies on views, and possibly completed views, which wouldn’t be measurable if the content was served from a local cache.
After unsuccessful struggling with storeurl.pl method , I was searching for alternate method to cache youtube videos. Finally I found ruby base method using Nginx to cache YT. Using this method I was able to cache all Youtube videos almost perfectly. (not 100%, but it works fine in most cases with some modification.I am sure there will be some improvement in near future).
Updated: 24thth August, 2012
Thanks to Mr. Eliezer Croitoru & Mr.Christian Loth & others for there kind guidance.
Following components were used in this guide.
Proxy Server Configuration:
Ubuntu Desktop 10.4
Nginix version: nginx/0.7.65
Squid Cache: Version 2.7.STABLE7
Client Configuration for testing videos:
Windows XP with Internet Explorer 6
Windows 7 with Internet Explorer 8
Lets start with the Proxy Server Configuration:
1) Update Ubuntu
First install Ubuntu, After installation, configure its networking components, then update it by following command
apt-get install update
2) Install SSH Server [Optional]
Now install SSH server so that you can manage your server remotely using PUTTY or any other ssh tool.
apt-get install openssh-server
3) Install Squid Server
Now install Squid Server by following command
apt-get install squid
[This will install squid 2.7 by default]
Now edit squid configuration files by using following command
nano /etc/squid/squid.conf
Remove all lines and paste the following data
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
# SQUID 2.7/ Nginx TEST CONFIG FILE
# Email: aacable@hotmail.com
# Web : https://aacable.wordpress.com
# PORT and Transparent Option
http_port 8080 transparent
server_http11 on
icp_port 0
# Cache is set to 5GB in this example (zaib)
store_dir_select_algorithm round-robin
cache_dir aufs /cache1 5000 16 256
cache_replacement_policy heap LFUDA
memory_replacement_policy heap LFUDA
# If you want to enable DATE time n SQUID Logs,use following
emulate_httpd_log on
logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
log_fqdn off
# How much days to keep users access web logs
# You need to rotate your log files with a cron job. For example:
# 0 0 * * * /usr/local/squid/bin/squid -k rotate
logfile_rotate 14
debug_options ALL,1
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
#[zaib] I used DNSAMSQ service for fast dns resolving
#so install by using "apt-get install dnsmasq" first
dns_nameservers 127.0.0.1 221.132.112.8
#ACL Section
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563 # https, snews
acl SSL_ports port 873 # rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 631 # cups
acl Safe_ports port 873 # rsync
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access allow all
http_reply_access allow all
icp_access allow all
#[zaib]I used UBUNTU so user is proxy, in FEDORA you may use use squid
cache_effective_user proxy
cache_effective_group proxy
cache_mgr aacable@hotmail.com
visible_hostname proxy.aacable.net
unique_hostname aacable@hotmail.com
cache_mem 8 MB
minimum_object_size 0 bytes
maximum_object_size 100 MB
maximum_object_size_in_memory 128 KB
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880
refresh_pattern . 0 50% 4320
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
# Youtube Cache Section [zaib]
url_rewrite_program /etc/nginx/nginx.rb
url_rewrite_host_header off
acl youtube_videos url_regex -i ^http://[^/]+\.youtube\.com/videoplayback\?
acl range_request req_header Range .
acl begin_param url_regex -i [?&]begin=
acl id_param url_regex -i [?&]id=
acl itag_param url_regex -i [?&]itag=
acl sver3_param url_regex -i [?&]sver=3
cache_peer 127.0.0.1 parent 8081 0 proxy-only no-query connect-timeout=10
cache_peer_access 127.0.0.1 allow youtube_videos id_param itag_param sver3_param !begin_param !range_request
cache_peer_access 127.0.0.1 deny all
Save & Exit.
4) Install Nginx
Now install Nginix by
apt-get install nginx
Now edit its config file by using following command
nano /etc/nginx/nginx.conf
Remove all lines and paste the following data
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
# This config file is not written by me, [syed.jahanzaib]
# My Email address is inserted Just for tracking purposes
# For more info, visit http://code.google.com/p/youtube-cache/
# Syed Jahanzaib / aacable [at] hotmail.com
user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_static on;
gzip_comp_level 6;
gzip_disable .msie6.;
gzip_vary on;
gzip_types text/plain text/css text/xml text/javascript application/json application/x-javascript application/xml application/xml+rss;
gzip_proxied expired no-cache no-store private auth;
gzip_buffers 16 8k;
gzip_http_version 1.1;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
# starting youtube section
server {
listen 127.0.0.1:8081;
location / {
root /usr/local/www/nginx_cache/files;
#try_files "/id=$arg_id.itag=$arg_itag" @proxy_youtube; # Old one
#try_files "$uri" "/id=$arg_id.itag=$arg_itag.flv" "/id=$arg_id-range=$arg_range.itag=$arg_itag.flv" @proxy_youtube; #old2
try_files "/id=$arg_id.itag=$arg_itag.range=$arg_range.algo=$arg_algorithm" @proxy_youtube;
}
location @proxy_youtube {
resolver 221.132.112.8;
proxy_pass http://$host$request_uri;
proxy_temp_path "/usr/local/www/nginx_cache/tmp";
#proxy_store "/usr/local/www/nginx_cache/files/id=$arg_id.itag=$arg_itag"; # Old 1
proxy_store "/usr/local/www/nginx_cache/files/id=$arg_id.itag=$arg_itag.range=$arg_range.algo=$arg_algorithm";
proxy_ignore_client_abort off;
proxy_method GET;
proxy_set_header X-YouTube-Cache "aacable@hotmail.com";
proxy_set_header Accept "video/*";
proxy_set_header User-Agent "YouTube Cacher (nginx)";
proxy_set_header Accept-Encoding "";
proxy_set_header Accept-Language "";
proxy_set_header Accept-Charset "";
proxy_set_header Cache-Control "";}
}
}
Save & Exit.
Now Create directories to hold cache files
mkdir /usr/local/www
mkdir /usr/local/www/nginx_cache
mkdir /usr/local/www/nginx_cache/tmp
mkdir /usr/local/www/nginx_cache/files
chown www-data /usr/local/www/nginx_cache/files/ -Rf
Now create nginx .rb file
touch /etc/nginx/nginx.rb
chmod 755 /etc/nginx/nginx.rb
nano /etc/nginx/nginx.rb
Paste the following data in this newly created file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
#!/usr/bin/env ruby1.8
# This script is not written by me,
# My Email address is inserted Just for tracking purposes
# For more info, visit http://code.google.com/p/youtube-cache/
# Syed Jahanzaib / aacable [at] hotmail.com
# url_rewrite_program <path>/nginx.rb
# url_rewrite_host_header off
require "syslog"
require "base64"
class SquidRequest
attr_accessor :url, :user
attr_reader :client_ip, :method
def method=(s)
@method = s.downcase
end
def client_ip=(s)
@client_ip = s.split('/').first
end
end
def read_requests
# URL <SP> client_ip "/" fqdn <SP> user <SP> method [<SP> kvpairs]<NL>
STDIN.each_line do |ln|
r = SquidRequest.new
r.url, r.client_ip, r.user, r.method, *dummy = ln.rstrip.split(' ')
(STDOUT << "#{yield r}\n").flush
end
end
def log(msg)
Syslog.log(Syslog::LOG_ERR, "%s", msg)
end
def main
Syslog.open('nginx.rb', Syslog::LOG_PID)
log("Started")
read_requests do |r|
if r.method == 'get' && r.url !~ /[?&]begin=/ && r.url =~ %r{\Ahttp://[^/]+\.youtube\.com/(videoplayback\?.*)\z}
log("YouTube Video [#{r.url}].")
"http://127.0.0.1:8081/#{$1}"
else
r.url
end
end
end
main
Save & Exit.
5) Install RUBY
What is RUBY?
Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. [syed.jahanzaib]
Now install RUBY by following command
apt-get install ruby
6) Configure Squid Cache DIR and Permissions
Now create cache dir and assign proper permission to proxy user
mkdir /cache1
chown proxy:proxy /cache1
chmod -R 777 /cache1
Now initialize squid cache directories by
squid -z
You should see Following message
Creating Swap Directories
7) Finally Start/restart SQUID & Nginx
service squid start
service nginx restart
Now from test pc, open youtube and play any video, after it download completely, delete the browser cache, and play the same video again, This time it will be served from the cache. You can verify it by monitoring your WAN link utilization while playing the cached file.
Look at the below WAN utilization graph, it was taken while watching the clip which is not in cache
WAN utilization of Proxy, While watching New Clip (Not in cache)
Now Look at the below WAN utilization graph, it was taken while watching the clip which is now in CACHE.
WAN utilization of Proxy, While watching already cached Clip
Playing Video, loaded from the cache chunk by chunk
It will load first chunk from the cache, if the user keep watching the clip, it will load next chunk at the end of first chunk, and will continue to do so.
Video cache files can be found in following locations.
/usr/local/www/nginx_cache/files
e.g:
ls -lh /usr/local/www/nginx_cache/files
The above file shows the clip is in 360p quality, and the length of the clip is 5:54 Seconds.
itag=34 shows the video quality is 360p.
Credits: Thanks to Mr. Eliezer Croitoru & Mr.Christian Loth & others for there kind guidance.
Find files that have not been accessed from x days. Useful to delete old cache files that have not been accessed since x days.
find /usr/local/www/nginx_cache/files -atime +30 -type f
Regard’s
Syed Jahanzaib
Howto redirect audio/video or some contents to another proxy using SQUID
Filed under: Linux Related — Tags: aacable, howto redirect contetns to another proxy, linux load balancing, proxy load balancing, redirect squid traffic, squid, Squid Proxy Load Balance — Syed Jahanzaib / Pinochio~:) @ 11:45 AM
6 Votes
Howto redirect Audio/Video or some contents based on extensions to another proxy using SQUID
Assalam Va Alaekum,
Following guide will show you how to redirect some contents from Squid proxy server to another proxy server (possibly squid or ISA, whatever). This is one of the oldest tricks in the book. People who are engaged in Linux after late 90’s knows about it very well. I used this scenario at my network in year 2003-2004 when I had SPEEDCAST IVS downlink of 256Kb and 64Kbs ISDN link. It was very useful at that time and user were really amazed with the browsing speed
Here is the test scenario, we have two proxy servers , proxy1 and proxy2.
PROXY1 is our master/parent proxy server connected with fast internet link and we want it to be reserved just for browsing and other important stuff, and we want that any request for .mp3 .exe .wmv .avi content must be redirect to PROXY2 server which is connected with lower bandwidth Link like satellite downlink or other.
Install 2 squid servers.
proxy1 = 192.168.2.1 [master/parent proxy]
proxy2 = 192.168.2.9 [for download redirected_content only]
Now on proxy1 (parent/master) , add following lines in /etc/squid/squid.conf
1
2
3
4
5
acl mynetwork src 0.0.0.0/0.0.0.0
acl redirect_content urlpath_regex -i \.MP3$ \.AVI$ \.WMV$ \.RM$ \.FLV$ \.ZIP$ \.RAR$ \.EXE$ \.mp3$ \.avi$ \wmv.$ \.rm$ \.flv$ \.zip$ \.rar$ \.exe$
cache_peer 192.168.2.9 parent 8080 3130
cache_peer_access 192.168.2.9 allow mynetwork redirect_content
never_direct allow redirect_content
All Done. Now test the setup. Set client browser proxy to use proxy1 and try to download anything (which is in redirect_content acl e.g rm or mp3 songs)
See the attached images.
apniisp- Testing Song Download from internet
Monitor Squid access log at proxy1 and proxy2, you will clearly see that only redirect_content are redirected to proxy2, rest of traffic is using proxy1 only.
PARENT PROXY1 Squid Access Log
PROXY2 Squid Access Log
This way you can do load balancing on your proxy servers.
The biggest advantage of such load balancing is that user browsing will not be effected if 20-30 users are simultaneously downloading any media contents or your marked contents
Cheers,
Allah Hafiz
Regard’s
--------
-