“DNS”目录存档

关于分布式负载均衡的几个设置

2008年01月29日,星期二

服务器对外的展现要统一,简洁。
即用户在URL里看到的地址是很方便记忆的。

而对于页面中嵌入的图片,脚本,样式表,视频,音频等文件,则可以放在其它服务器上。

第一层: DNS记录轮询负载均衡
www.google.com为例进行分析
1.首先是将 www.google.com,指定多个cname,根据不同的网络线路请求得到不同的别名

比如在中国得到的别名为:www.l.google.com,
www.l.google.com 的别名为: www-china.l.google.com

www-china.l.google.com再指定多个ip的a记录,
例如 ip 64.233.189.99, 64.233.189.104,64.233.167.147.

99,104,147这三个ip分别实现ip别名绑定,即如果99这个ip挂了,104会自动接管该ip到本机。

通过这样的方式,保证用户最终访问到的ip始终在线。

在cname解析的地方,可以使用多层cname解析,以实现更细粒度的负载均衡并灵活切换。

假设当www-china.l.google.com下整个节点都断网了,则只需要将www.l.google.com指向www-usa.l.google.com这样就可以切换了。

(纯属猜测:指定了cname的域名,则不要再配置a记录,因为a记录的优先级比cname高。)

dns解析过程记录如下:

[code]
E:\>nslookup
DNS request timed out.
timeout was 2 seconds.
*** Can't find server name for address 192.168.18.1: Timed out
*** Default servers are not available
Default Server:  UnKnown
Address:  192.168.18.1

> server 202.96.128.86
DNS request timed out.
timeout was 2 seconds.
Default Server:  [202.96.128.86]
Address:  202.96.128.86

> server 202.96.128.86
Default Server:  cache-a.guangzhou.gd.cn
Address:  202.96.128.86

> set q=cname
> www.google.com
Server:  cache-a.guangzhou.gd.cn
Address:  202.96.128.86

Non-authoritative answer:
www.google.com  canonical name = www.l.google.com
> www.l.google.com
Server:  cache-a.guangzhou.gd.cn
Address:  202.96.128.86

Non-authoritative answer:
www.l.google.com        canonical name = www-china.l.google.com
> www-china.l.google.com
Server:  cache-a.guangzhou.gd.cn
Address:  202.96.128.86

l.google.com
primary name server = f.l.google.com
responsible mail addr = dns-admin.google.com
serial  = 1331146
refresh = 900 (15 mins)
retry   = 900 (15 mins)
expire  = 1800 (30 mins)
default TTL = 60 (1 min)
> set q=a
> www-china.l.google.com
Server:  cache-a.guangzhou.gd.cn
Address:  202.96.128.86

Non-authoritative answer:
Name:    www-china.l.google.com
Addresses:  64.233.189.99, 64.233.189.104

[/code]

(也可用这个命令查询: dig @202.96.128.86 www.google.com +trace)

第二层,服务端请求分发。

前端的ip得到客户端的请求,并不是在本机进行业务逻辑处理,而是对请求进行初步解析过滤,再分发给其它服务器进行处理,
请求分发规则通常是基于url的,也可根据其它附加条件进行分发,例如GET/POST,user-Agent,remoteAddr,urlhash等。

第三层:客户端分发请求。

通常用于对图片资源的请求进行分发,按找浏览器的默认限制,对同一服务器的并发连接不超过2个,
因此,假设一个网页里要显示40张图片,而这40张图片,如果使用同一个域名的话,及时后台做了请求分发,
而受浏览器的限制,假设一秒中下载两张图,打开这40张图片,也需要20秒,

而在采取客户端分发请求的模式,将图片的链接自动分布的请求到远程多台服务器,那么以同时请求5台为例,则相当于将对网站的并发请求提高了 5倍,
获得了了10个并发请求。
不光是图片资源,其它资源也都可以采取这种模式,url的动态分布,可以在服务端生成html代码的时候完成,
也可以在html的Javascript中预先存放一个可用服务器列表

www.flickr.com为例:

http://farm1.static.flickr.com/

http://farm2.static.flickr.com/

http://farm3.static.flickr.com/

……

http://farmx.static.flickr.com/

Tags: DNS, javascript, 负载均衡, 集群

Related posts

深圳电信的http://219.133.33.37/update/step1.aspx?p

2007年12月15日,星期六

深圳电信对 ADSL上网的 DNS 劫持很过分,用firefox打开网页,每天晚上上一会儿网,就几乎几十次的出现这玩意,

总要去打开 219.133.33.37 的东西,无语!!!

类似情况可参考:

http://www.williamlong.info/archives/1116.html

[code]

<html>
<head>
<meta http-equiv="pragma" content="no-cache"/>
<meta http-equiv="Cache-Control" content="no-store,no-cache,must-revalidate,post-check=0,pre-check=0,max-age=0"/>
<meta http-equiv="expires" content="-1"/>
<meta http-equiv="refresh" content="2"/>
</head>
<body>
<iframe src="about:blank" width="0" height="0" frameborder="0" style="display:none"></iframe>
<script language="javascript">
window.frames[0].location = "http://219.133.33.37/update/step1.aspx?p=" +
"sz10000@163.gd|" +
Math.floor((new Date()).getTime()/1000) + "|" +
navigator.appMinorVersion + "|" +
screen.availHeight + "|" +
screen.availWidth + "|" +
screen.colorDepth + "|" +
screen.height + "|" +
screen.width;
</script>
</body>
</html>

[/code]

拼出来的url 为: http://219.133.33.37/update/step1.aspx?p=sz10000@163.gd|1197733344|;SP2;|994|1280|32|1024|1280

不过被 Firefox的 noScript被挡掉了.

Tags: 219.133.33.37, DNS, FireFox, 深圳电信

Related posts

在Tomcat 6.0.14中安装配置awstats

2007年12月7日,星期五

在Tomcat 6.0.x中配置awstats
需要下载的文件有:

perl : http://downloads.activestate.com/ActivePerl/Windows/5.8/ActivePerl-5.8.8.822-MSWin32-x86-280952.msi
来源: http://www.activestate.com/store/activeperl/download/

http://update.cz88.net/soft/qqwry.rar

http://awstats.sourceforge.net/files/awstats-6.8.tar.gz

geoip插件相关:

http://www.maxmind.com/download/geoip/api/pureperl/Geo-IP-PurePerl-1.18.tar.gz

http://www.maxmind.com/download/geoip/database/GeoIP.dat.gz

http://www.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

http://www.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz

安装过程可参考

参考: http://blog.zhangjianfeng.com/article/317
PurePerl.pm的安装可参考:

http://topic.csdn.net/u/20070323/11/903dd121-10d8-42d2-b4eb-a6cb60a22e8a.html

其它配置可参考:

http://www.chedong.com/blog/archives/000410.html

http://blogger.org.cn/blog/more.asp?name=chenjiejacky&id=20464

安装perl之后,确认环境变量里PATH有perl的bin目录
Path=E:\oracle\product\10.2.0\client_1\bin;D:\Perl\bin;
D:\Java\jdk1.6.0_03\bin;E:\oracle\product\10.2.0\db_1\bin;
C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;
C:\Program Files\Microsoft SQL Server\80\Tools\BINN

1.为了控制查看awstats的权限,我在
E:\apache-tomcat-6.0.14\conf\tomcat-users.xml的内容配置为:

<?xml version=’1.0′ encoding=’utf-8′?>
<tomcat-users>
<role rolename=”awstats”/>
<user username=”awstats” password=”awstats” roles=”awstats”/>
</tomcat-users>

2.由于权限限制

E:\apache-tomcat-6.0.14\webapps\awstats\META-INF\context.xml,内容为:

<!– tomcat 6.0 need privileged is true for CGIServlet –>
<Context reloadable=”true” privileged=”true” >

</Context>

否则启动的时候会出下面的错误信息:

[ERROR]main-org.apache.catalina.startup.HostConfig-Error deploying web application directory awstats
java.lang.SecurityException: Servlet of class org.apache.catalina.servlets.CGIServlet is privileged and cannot be loaded by this web application
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1134)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:981)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4045)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4351)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:920)
at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:883)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1138)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
at org.apache.catalina.core.StandardService.start(StandardService.java:516)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:566)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
3.配置web.xml,增加权限控制
web.xml的内容为:

<?xml version=”1.0″ encoding=”UTF-8″?>
<web-app xmlns=”http://java.sun.com/xml/ns/j2ee” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd” version=”2.4″>
<servlet>
<servlet-name>cgi</servlet-name>
<servlet-class>org.apache.catalina.servlets.CGIServlet</servlet-class>
<init-param>
<param-name>clientInputTimeout</param-name>
<param-value>100</param-value>
</init-param>
<init-param>
<param-name>debug</param-name>
<param-value>0</param-value>
</init-param>
<init-param>
<param-name>cgiPathPrefix</param-name>
<param-value>WEB-INF/cgi-bin</param-value>
</init-param>
<load-on-startup>5</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>cgi</servlet-name>
<url-pattern>/cgi-bin/*</url-pattern>
</servlet-mapping>
<!– Define reference to the user database for looking up roles –>
<resource-env-ref>
<description>
Link to the UserDatabase instance from which we request lists of
defined role names. Typically, this will be connected to the global
user database with a ResourceLink element in server.xml or the context
configuration file for the Manager web application.
</description>
<resource-env-ref-name>users</resource-env-ref-name>
<resource-env-ref-type>org.apache.catalina.UserDatabase</resource-env-ref-type>
</resource-env-ref>
<!– Define a Security Constraint on this Application –>
<security-constraint>
<web-resource-collection>
<web-resource-name>awstats status</web-resource-name>
<url-pattern>/cgi-bin/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>awstats</role-name>
</auth-constraint>
</security-constraint>
<!– Define the Login Configuration for this Application –>
<login-config>
<auth-method>BASIC</auth-method>
<realm-name>Awstats status,username: awstats,password: awstats</realm-name>
</login-config>
<!– Security roles referenced by this web application –>
<security-role>
<description>
The role that is required to log in to the Manager Application
</description>
<role-name>awstats</role-name>
</security-role>
<welcome-file-list>
<welcome-file>index.jsp</welcome-file>
<welcome-file>index.html</welcome-file>
<welcome-file>index.htm</welcome-file>
</welcome-file-list>
</web-app>

4.调整文件夹目录

E:\APACHE~1.14\webapps\awstats 的目录

007-12-07 18:14 <DIR> .
007-12-07 18:14 <DIR> ..
007-12-07 08:48 <DIR> classes
007-12-07 08:48 <DIR> css
007-12-07 08:48 <DIR> docs
007-12-07 08:48 <DIR> icon
007-12-07 08:49 211 index.jsp
007-12-07 08:48 <DIR> js
007-12-07 08:56 <DIR> logs
007-12-07 08:48 <DIR> META-INF
007-11-24 22:05 6,721 README.TXT
007-12-07 08:47 <DIR> WEB-INF

E:\APACHE~1.14\webapps\awstats\WEB-INF 的目录

2007-12-07 08:47 <DIR> .
2007-12-07 08:47 <DIR> ..
2007-12-07 08:56 <DIR> cgi-bin
2007-12-07 08:47 <DIR> tools
2007-12-07 08:42 2,389 web.xml

E:\APACHE~1.14\webapps\awstats\WEB-INF\cgi-bin 的目录

007-12-07 08:56 <DIR> .
007-12-07 08:56 <DIR> ..
006-02-09 06:26 5,407 awredir.pl
007-12-07 08:56 2,343 awstats.localhost.conf
007-12-07 08:55 2,340 awstats.localhost.conf.bak
007-11-24 03:18 60,544 awstats.model.conf
007-11-24 03:18 556,625 awstats.pl
007-12-07 08:56 13,569 awstats122007.localhost.txt
006-07-27 09:37 1,847 awstatsext.conf
007-12-07 08:56 678 dnscachelastupdate.localhost.hash
007-12-03 12:18 1,079,842 GeoIP.dat
007-09-13 04:08 2,020,543 GeoIPASNum.dat
007-12-03 16:25 25,914,398 GeoLiteCity.dat
007-12-07 08:48 <DIR> lang
007-12-07 08:48 <DIR> lib
007-03-15 14:35 184 perl插件安装说明.txt
007-12-07 08:54 <DIR> plugins
007-12-05 17:58 6,812,739 QQWry.Dat

E:\APACHE~1.14\webapps\awstats\WEB-INF\cgi-bin\plugins 的目录

2007-12-07 08:54 <DIR> .
2007-12-07 08:54 <DIR> ..
2006-02-09 06:26 3,682 clusterinfo.pm
2006-02-09 06:26 2,960 decodeutfkeys.pm
2007-12-07 08:48 <DIR> example
2007-12-07 08:54 <DIR> geo
2006-05-06 11:51 7,203 geoip.pm
2006-02-09 06:26 3,736 geoipfree.pm
2006-09-04 15:04 112,953 geoip_city_maxmind.pm
2006-05-06 11:54 16,885 geoip_isp_maxmind.pm
2006-05-06 11:54 16,919 geoip_org_maxmind.pm
2006-05-06 11:54 22,283 geoip_region_maxmind.pm
2004-01-01 05:41 5,074 graphapplet.pm
2006-02-09 06:26 5,206 hashfiles.pm
2005-02-19 21:50 7,169 hostinfo.pm
2006-02-09 06:26 2,439 ipv6.pm
2005-04-11 05:08 2,501 qqhostinfo.pm
2005-03-28 11:29 7,603 qqwry.pl
2006-02-09 06:26 4,689 rawlog.pm
2006-02-09 06:26 1,890 timehires.pm
2006-02-09 06:26 2,954 timezone.pm
2006-07-30 03:20 8,212 tooltips.pm
2006-02-09 06:26 4,799 urlalias.pm
2006-02-09 06:26 3,616 userinfo.pm

我的awstats.localhost.conf配置如下:

Include=”common.conf”
PageCode=”GB2312″ # UTF-8
LogFile=”E:/apache-tomcat-6.0.14/logs/access.log”
SiteDomain=”618119.com”
LoadPlugin=”decodeutfkeys”
LoadPlugin=”hashfiles”
LoadPlugin=”geoip”
LoadPlugin=”tooltips”
LoadPlugin=”qqhostinfo”
LoadPlugin=”geoip_city_maxmind GEOIP_STANDARD GeoLiteCity.dat”
LoadPlugin=”geoip_org_maxmind GEOIP_STANDARD GeoIPASNum.dat”
LogFormat=1
HostAliases=”localhost 618119.com www.618119.com
DirIcons=”/awstats/icon”
DNSLookup=1
DirData=”.”
DirCgi=”/cgi-bin”
AllowToUpdateStatsFromBrowser=1

Include=”awstatsext.conf”

一个非常有用的参考地址: http://www.antezeta.com/awstats.html

扩展设置:
# form http://www.antezeta.com/awstats.html

ExtraSectionName1=”Google Searches – Top 50″
ExtraSectionCodeFilter1=”200 304″
ExtraSectionCondition1=”REFERER,(.*www\.google.*)”
ExtraSectionFirstColumnTitle1=”Search”
ExtraSectionFirstColumnValues1=”REFERER,p=([^&]+)||REFERER,q=([^&]+)||REFERER,as_p=([^&]+)||REFERER,as_q=([^&]+)”
ExtraSectionFirstColumnFormat1=”<a href=’http://www.google.cn/search?q=%s’ title=’Click to execute search’>%s</a>”
ExtraSectionStatTypes1=PHBL
ExtraSectionAddAverageRow1=0
ExtraSectionAddSumRow1=1
MaxNbOfExtra1=50
MinHitExtra1=1

ExtraSectionName2=”Google crawls – Top 50″
ExtraSectionCodeFilter2=”200 304″
ExtraSectionCondition2=”UA,(.*Googlebot.*)”
ExtraSectionFirstColumnValues2=”URL,(.*)”
ExtraSectionFirstColumnFormat2=”<a href=’http://618119.com%s’ title=’Item Crawled’>%s</a>”
ExtraSectionStatTypes2=PHBL
ExtraSectionAddAverageRow2=0
ExtraSectionAddSumRow2=1
MaxNbOfExtra2=50
MinHitExtra2=1

ExtraSectionName3=”sitemap.xml.gz downloads by Useragent”
ExtraSectionCodeFilter3=”200 304″
ExtraSectionCondition3=”URL,(^\/sitemap\.xml\.gz)”
ExtraSectionFirstColumnTitle3=”UA”
ExtraSectionFirstColumnValues3=”UA,(.*)”
ExtraSectionStatTypes3=HBL
ExtraSectionAddAverageRow3=0
ExtraSectionAddSumRow3=1
MaxNbOfExtra3=10
MinHitExtra3=1

ExtraSectionName4=”Referring Sites by domain – Top 25″
ExtraSectionCodeFilter4=”200 304″
# Filter on ANY REFERER except “mysite”. Change mysite to your domain name.
ExtraSectionCondition4=”REFERER,^(?!http:\/\/www\.618119\.com)”
ExtraSectionFirstColumnTitle4=”Site”
ExtraSectionFirstColumnValues4=”REFERER,^[hH][tT][tT][pP]:\/\/([^\/]+)\/”
ExtraSectionFirstColumnFormat4=”<a href=’http://%s/’ rel=’nofollow’ title=’http://%s/ [new window]‘>%s</a>”
ExtraSectionStatTypes4=PHL
ExtraSectionAddAverageRow4=1
ExtraSectionAddSumRow4=1
MaxNbOfExtra4=25
MinHitExtra4=1

ExtraSectionName5=”Top 30 RSS Readers/Spiders”
ExtraSectionCodeFilter5=”200 304″
ExtraSectionCondition5=”URL,\.xml|\.rdf|\.rss|\/feed”
ExtraSectionFirstColumnTitle5=”RSS Reader/Spider”
ExtraSectionFirstColumnValues5=”UA,(.*)”
ExtraSectionStatTypes5=HBL
ExtraSectionAddAverageRow5=1
ExtraSectionAddSumRow5=1
MaxNbOfExtra5=30
MinHitExtra5=1

ExtraSectionName6=”Downloads (diff,doc,pdf,rtf,sh,tgz,zip) – Top 10″
ExtraSectionCodeFilter6=”200 304″
ExtraSectionCondition6=”URL,(.*((\.diff)|(\.doc)|(\.pdf)|(\.rtf)|(\.sh)|(\.tgz)|(\.zip)))”
ExtraSectionFirstColumnTitle6=”Download”
ExtraSectionFirstColumnValues6=”URL,(.*)”
ExtraSectionFirstColumnFormat6=”%s”
ExtraSectionStatTypes6=HBL
ExtraSectionAddAverageRow6=0
ExtraSectionAddSumRow6=1
MaxNbOfExtra6=10
MinHitExtra6=1

Tags: 618119.com, Apache, Apache Tomcat, awstats, perl

Related posts