Case Study on Oracle Web Cache and the Edge Side Includes (ESI)
De norske Bokklubbene
- Company background ? history, business, revenue, number of users/customers etc
- What does De Norske Bokklubbene do?
De norske Bokklubbene translates to ?The Norwegian Bookclubs?. We sell books through 20 or so clubs. There is also an international handler club to handle books that are not published by Bokklubbene or associated companies. We are an amazon equilivalent in norway with a large membership across the country.
- A little bit of history about Bokklubbene will be helpful.
Norway?s major book club operation established 1961. More than 700 000 memberships total. De norske Bokklubbene runs fourteen book clubs, an art club and an internet book store. More than 25 % of the Norwegian households are members of our book clubs and we distribute nearly 50% fiction sold in Norway De norske Bokklubbene presents the world?s leading authors to Norwegian children, youth and adults.
- How many visitors does Bokklubbene?s web site get each day or each week?
Our website receives upto 100 to 300 hits per second. More than 200,000 members visit wwww.bokklubbene every month.
- Business problem that De Norske Bokklubbene is trying to solve
- For example, Bokkluene needs a fast web site for its members to share their book review, etc.
The current project was to rewrite the infrastructure behind the website. Previously the international handler club was run on a very different solution. Both solutions were propreitary, and not handling the load and required customisations well over time. The new solution was to create a common solution written from scratch with Java.
Application Architecture & Oracle WebCache
- Technology requirement to resolve the business problem
- For example, the solution must be able to support 4000 concurrent users or offers an average time of 2 seconds per page.
Unification of solutions, stability and customisation were the key goals. The previous solution (SilverStream and iStore) required the application servers to be regularly restarted. They also involved a lot of work to make small customisations to the website.
- Selection process on which technology solution is most suitable (e.g. evaluation of other mid-tier vendors or caching appliances); Why Oracle?
- What criteria does Bokkluene have in choosing the technology vendor? Scalability, performance, completeness in functionality, etc.
The key selection was to use Java rather than a higher level custom solution. By writing our own solution to suit our own needs. Choices followed on from this. Struts, and related jakarta tools, were chosen because of their MVC design and simplicity of solution. Opensource tools were also favoured for their better support via their communities. Bokklubbene's databases run on Oracle, so for simplicity of support and integration Oracle10g AS was chosen as the application server. Oracle WebCache was not an initial need to the project. But due to having millions of books in the database, and complex searches being a potential bottleneck to the application server, WebCache was keep in mind as a potential solution.
During development WebCache became more inticing for a number of reasons. All pictures are stored in the database and hence are processed by the application server. This is really an unneccessary load for the application server. By using WebCache all pictures could be pre-fetched to prevent this additional load during normal usage. The same logic applied to many of the websites page fragments that were static. WebCache's ability to prevent DoS attacks and problems was also seen as a important part to the new solution. The caching of so much of the website is an important mean to ensure the application servers are never overloaded.
ESI Include VS Inline tags, & caching rules…
- Detailed description of the technology solution
- Describe the application
- How is the application generated? JSP?
Struts and Tiled JSP. Behind this there is also a lot of Xslt happening. This is usually for the editorial parts of the page.
WebCache esi:inline tags are used. During the evaulation, despite Oracle's recommendations, it was seen that esi:inline was benefical over esi:include. With the include tags core view logic/infrastructure would be processed by the WebCache. The solution could only be run with WebCache. This had a number of disadvantages, it would mean that developers would not be able to run OC4J on their machines without also running WebCache. For development and testing purposes this was deemed unnacceptable. Using include also tied the infrastructure to Oracle WebCache or Squid 4, and we did not desire this vendor lock-in. With esi:inline tags, we could simply embed the tag information into our existing html output. There was also no advantage to use include tags over inline tags as inline tags can be made to be fetchable as well. For those fragments that have short cacheable periods struts actions were added to return just the html fragment required. This approach has proven to be hugely successful as it could be developed after the design stage by analysing what fragments were timing out the most often and hence requiring webcache to fetch the whole page again, and corrected. This approach to performance tuning is favoured. It also keep the design of our solution completely within the MVC bounds of the Java code.
Each page contains from 10 to 20 fragments over an heirarchy upto 4 levels deep.
- Which part of the application is cacheable or which part is not?
Everything is cacheable except for customer pages, shopping cart information, login name information, numbers of books available, and the campaigns to joing a club. Snippets of information within an otherwise cacheable page (eg number of books available, customers discount price for that book, the customers name, or the shopping cart information) was solved by embedding this non-cacheable data inside an iframe. This way the page was cacheable but had little holes in it (iframes) that the noncacheable data would appear in.
- Do you build the application with JESI tags in mind or decide to add them afterwards?
JESI tags are not used because they only support esi:include. The inline tags were added afterwards.
- How do you control invalidation?
Through max-age attributes to esi:inline tags. And with cacheability rules in webcache manager (webcache.xml).
Invalidation requests are also triggered off database changes via plsql procedures. The same plsql procedure is used during redeployment of the application at the application server level.
- Which development tool do you use (e.g. JDeveloper)? Did you use the ESI servlet filter extension in Jdeveloper?
We use NetBeans 4.0. There are a number of features NetBeans provides or handles better than Jdeveloper. Writing inline tags is simple enough to be accomplished in any text editor.
Here is our top layout jsp. It illustrates the first level of esi:inline usage.
- Diagrams of the flow of application will be helpful too (if possible)
(hassle me about this one. Just not enough time today.)
- Deployment architecture
- What does the system architecture look like? Any firewalls and hardware load balancers
Two Firewalls. First completely around company. Second between Application servers and database. There is a cisco load balancer infront of the webcache machines. It is only used to detect failure and to redirect traffic to a working server in such a situation.
- _How many instances of Web Cache? Are they in a cluster?
Two WebCache machines clustered. The second is used for failover.
- How many instances of iAS mid-tiers (OHS, OC4J, etc.)? And database?
Three Application Servers (2x 2.5GHz cpu, 3Gb RAM), due to the WebCache solution these never reach 5% cpu usage. One database server (4x 2.5GHz cpu, 8Gb RAM).
- What is the network speed? T1 line or 100Mbits? Are your end-users limited by bandwidth?
- Staging and Production environment
- What kind of load test do you run?
Apache benchmark 2 (ab2) and recursive wget.
- Performance results ? throughput, peak load in terms of # of hits on the website, amount of cacheable content
ab2 -n 10000 -c 1
Requests per second: 64.11 #/sec (mean)
Time per request: 15.599 ms (mean)
where 80% took under 15ms to serve.
ab2 -n 10000 -c 100
Requests per second: 285.45 #/sec (mean)
Time per request: 35.033 ms (mean)
where 50% took under 27ms to serve.
Requests Served Since Start
Total Requests Served 1787185
Average Requests Served 156 /sec
Fresh Hits 89 %
Stale Hits 0 %
Cacheable Misses 1 %
Noncacheable Misses 10 %
Refreshes 0 %
Compressed Hits 0 %
Compressed Misses 6 %
- Challenges and lessons learned
- Clustering improved stability and performance dramatically.
- Regardless of how fast your application servers server, WebCache provides an simple level of DoS defense only improving stability for Application Server.
- Using esi:inline over esi:include tags allows the website to run without WebCache, avoiding any vendor lock-in, and allowing developers to use only OC4J on their personal machines. It also allows Squid 4 servers on the internet to cache your website according to the same rules you use.
- Using esi:inline and WebCache allows fragment and rule development to occur during the performance analsys stage (late) in the development cycle, and to target the most critical performance problems first.
- Resource and security limits must be increased. See /etc/limits.conf (or /etc/security/limits.conf).
- WebCache cannot run on Vmware. Unknown problems but completely unstable.