Agile Web Operations

Pimp my Website: Downsizing your Web Pages

Dan Ackerson — Sun, 16 Nov 2008 06:15:22 +0000

In the current economic downturn, everyone's keeping an extra close eye on the bottom line and looking for ways to save money. Let's compare the bytes of data on your network to water droplets and your website to a leaky faucet. You know you should take an afternoon and fix the damn thing, but something more important always comes up. But this doesn't stop the monthly bill from coming, does it? Holy cow, those drops add up! Today, I'd like to share a few tips on how you can get this faucet fixed up to save on monthly bandwidth fees.

HTML Compression

There are a lot of different ways you can reduce the size of your served content. Perhaps the easiest (and most important) is using a server side compression library like mod_gzip (for Apache 1.x servers) or mod_deflate (for Apache 2.x). All modern browsers can identify and reconstitute compressed content and both of these libraries allow you to set the default compression algorithm (between fast and best compressed). An average compression rate is about 70% so if you're currently paying any monthly overage fees to your hosting provider, check this out immediately.

Image Compression

Images used to be trickier. You'd have to play around with some locally installed picture editor trying to figure out which file type and which compression left the image pretty much the same but substantially reduced its size. Now, you can just head over to Smush.it and upload an image from your desktop or enter a URL of an online image for "smushing". In my initial Website Performance Optimization post, I hesitantly added 5 fairly big images. While trying to keep their sizes as small as possible yet still be understandable, one pic 'pingdom.png' still weighed in at a hefty 52KB. After "Smushing it" :

Pretty cool! But then I dug around some more and found the Online Image Optimizer site from Dynamic Drive. Just for fun, I fed the the above image into it to see what it could do :

Wow! I can go from 30KB down to 6KB for a similar image (and if you right click on this image above, you'll see that I did so). Great to be able to see in-line all the results of the various algorithms (and gives you a sneak-peek into how these tools work as well).

CSS & JS Compression

Depending on how desperate you are, you can even go as far as removing whitespace from large CSS and javascript files. There are lots of tools to help you with this but remember to do this at deploy time (so as not to make these files unmaintainable).

Even more hardcore, renaming (essentially obfuscating) your CSS class names to save on bytes transferred over the wire. Why specify <div id="border_top_frame" class="input_blackout_field"> when <div id="btf" class="ibf"> will do? Initially, the 30 byte savings may not seem like a lot. But if you're using this to render every row in a 100 record dataset, you've just shaved 3K off the page size. Again, only do this obfuscation step as part of a pre-production deployment to your web servers.

To recap:

HTML : mod_gzip or mod_deflate

images : Smush.it and Online Image Optimizer from Dynamic Drive

CSS & JS : whitespace remover and obfuscation enhancement

Content Delivery Network (CDN)

Why transfer the bytes yourself at all if you don't have to? Google offers free hosting for the latest versions of many, well-known scripts like jQuery, Prototype and script_aculo_us. Take advantage of this! A lot of these requests will even be automatically routed to the closest regional cache (for improved download speeds).

Depending on how large an audience you have, it might be time to step up to the big leagues. Akamai, BitGravity, Limelight Networks, or Panther Express all cost money but can improve end-user response times by 20% or more by serving up your static content from their huge, regionalized caches.

Cookies

The bane of privacy trumpeters, cookies are the de-facto standard for interacting with most popular websites nowadays. While standards are a good thing, abuses still run rampant especially in the sheer size of some of these cookies. Going through my cookies, I stumbled across this 1KB monster from komtrack.com:

<TDT><USERID>abcdef1234567000</USERID><CLICKLOGID>abcdef123z456000</CLICKLOGID><VIEWLOGID>0</VIEWLOGID><IMPRLOGID>0</IMPRLOGID><CLICK_PARTNER>17</CLICK_PARTNER><VIEW_PARTNER>0</VIEW_PARTNER><CLICK_PT></CLICK_PT><CLICK_BLOCK>1</CLICK_BLOCK><CLICK_PP1>1</CLICK_PP1><CLICK_PP2>0</CLICK_PP2><CLICK_PP3>0</CLICK_PP3><CLICK_PP4>0</CLICK_PP4><CLICK_PP5>0</CLICK_PP5><CLICK_PP6>0</CLICK_PP6><CLICK_PP7>0</CLICK_PP7><CLICK_PP8>0</CLICK_PP8><CLICK_PP9>0</CLICK_PP9><CLICK_TP1>6</CLICK_TP1><CLICK_TP2>0</CLICK_TP2><CLICK_TP3>2</CLICK_TP3><CLICK_TP4>3</CLICK_TP4><CLICK_TP5>0</CLICK_TP5><CLICK_TP6>0</CLICK_TP6><CLICK_TP7>0</CLICK_TP7><CLICK_TP8>1</CLICK_TP8><CLICK_TP9>0</CLICK_TP9><CLICK_TP10>0</CLICK_TP10><VIEW_PP1>0</VIEW_PP1><VIEW_PP2>0</VIEW_PP2><VIEW_PP3>0</VIEW_PP3><VIEW_PP4>0</VIEW_PP4><VIEW_PP5>0</VIEW_PP5><VIEW_PP6>0</VIEW_PP6><VIEW_PP7>0</VIEW_PP7><VIEW_PP8>0</VIEW_PP8><VIEW_PP9>0</VIEW_PP9><VIEW_TP1>0</VIEW_TP1><VIEW_TP2>0</VIEW_TP2><VIEW_TP3>0</VIEW_TP3><VIEW_TP4>0</VIEW_TP4><VIEW_TP5>0</VIEW_TP5><VIEW_TP6>0</VIEW_TP6><VIEW_TP7>0</VIEW_TP7><VIEW_TP8>0</VIEW_TP8><VIEW_TP9>0</VIEW_TP9><VIEW_TP10>0</VIEW_TP10><CP1>0</CP1><CP2>0</CP2><MOD></MOD></TDT>

An entire XML document as a cookie - nice! No clue how often this had to be transferred back and forth but its definitely a great example of bad cookie practice.

Another idea - if you use multiple domains for serving content, you can ensure that cookies are tied to only one of these domains This will free your static requests (images, javascript files, etc.) from cookie overhead. Keep your cookies lean and mean and your users will feel the difference.

You now have a few extra tricks up your sleeve to reduce your website's overall bandwidth utilization and reduce your monthly hosting bill. Speaking of which, when's the last time you actually reviewed this? Take a look at your monthly bandwidth limit, extra fee per GB overage and ratio of uploads to downloads. Does your hosting provider offer a contact that better fits your site's usage? Remember, everything is negotiable! You could save yourself a load of cash without tweaking a single configuration file.

I want to start agile, how can I do that?

Matthias Marschall — Thu, 13 Nov 2008 19:43:02 +0000

Last week a new colleague of a former colleague of mine ;-) asked me: "How can I get agile started? I really like the ideas but don't know where to begin or how to convince others around me to try it out. What should I do?"

Getting started with agile is not easy. First, you have to convince everybody that it's a good idea to try it and then, if you really managed to get the ball rolling, you begin to uncover all the hidden problems which have existed since decades in your development processes. Ouch. Only if you make it through these two phases will you be blessed with the advantages of an agile process: high speed, high quality, high visibility and high value of what you are doing.

Today I want to show you one way to tackle the first hurdle.

Identify the Biggest Problem

When I struggled to introduce agile processes and practices within my former organization, the single most useful tool which enabled my break through was a Value Stream Map as described by Marry and Tom Poppendieck in their first book about Lean Software Development.

I used the most simple version like the following picture shows:

The green lines are productive time spent to create value. The red lines represent the time that value sits around your office (not released to your customers) - wasted time. The x-axis shows the timeline. Just doing this simple exercise shows how badly out of proportion your value-added time is compared to wasted time. If you want to speed up feature delivery, you have to minimize the time wasted. There are a lot of red line segments, each of them pointing out an area of possible improvement. But you should focus your resources. Instead of trying to attack all problems at once you should only attack the biggest one. And the biggest problem is your biggest red line segment.

In my case, it was the waiting time from a feature being ready for release until the next release. We only did two releases a year so this waiting time could be close to six months in worst case. Compare that to the couple of days between development and testing and its pretty obvious which problem needs to be addressed first.

Attack the Biggest Problem

With such a Value Stream Map in hand you should be able to get the attention of your co-workers: "What? We really take up to two and a half years to deliver a feature :-O ?" Yes, we do. It wasn't nice to be forced to accept that truth, but it set the stage for change. After identifying our 6 month release cycles as the biggest waste, it was rather easy to come up with an initial idea: Let's do releases more often.

First, we decided to try to release something every two months. This was a classical "stretch goal" - a goal you can only reach by substantially changing the way you work because you'd never be able to attain it if you keep working like today. That need for a substantial, innovative change of our procedures gave us the required kick-start for agile.

Speeding up the process showed us one bottleneck after the other: Manual regression testing wasn't possible anymore as there wasn't enough time for it. So, we had to introduce automated tests. Month long design phases creating "perfect specs" were not possible anymore either. So we introduced user stories and acceptance test criteria. And so on. The Agile Train was rolling...

What are your experiences with introducing agile practices and processes? Did it work out well? How did you get it started? We would love to hear your story in the comments.

Pimp my Website: Streamlining Your Content

Dan Ackerson — Sun, 09 Nov 2008 06:15:09 +0000

Photo by ansik

Hopefully, you have a better idea about how (un)performant and (un)optimized your website is after going through some of the tools I introduced you to last week. You've made the first step in getting to know more about how your site is really served to your users. Now, I'll start walking you through some of the basic things you can do to start tuning up your website.

Ever caught yourself saying "yeah, I know the homepage is a bit heavy but the users only have to load it once - after that, it'll be cached and much faster". Well, according to Tenni Theurer, last year roughly half of the visitors that come through Yahoo! had an empty cache - meaning they have to download all the images and scripts on the page. This means that even if your page is viewed as often as Yahoo!'s, a lot of your users will be having to download all the content on a daily basis. Even if you're lucky enough to earn more than one page view per day, you should definitely be streamlining your homepage for the long haul. After all, you want these users to come back, right?

Minimize HTTP Requests

The first step, according to Yahoo!'s "Best Practices for Speeding Up Your Web Site" is minimizing the number of HTTP requests made by the users' browser to your web server(s). What we are basically trying to eliminate here is overhead. For every request and response (and all the packets in between) the TCP-IP stack adds a number of bytes which help make this protocol so reliable. As bytes add up (and browsers can only download so many files in parallel), we're looking to save some overhead and make the request and response handshake as efficient as possible.

A few of the ways to do this include :

combining separate javascript files at deploy time: multiple files can be concatenated into one, saving request overhead

combining separate CSS files at deploy time: same idea as above

combining images into a single file: either in an image map or CSS sprite

Don't know what a CSS sprite is? It's actually quite "old school technology" used in the early 8-bit and 16-bit game development era. Basically, take all your image files, stores them in a single file (background-image) and reference their (x, y) grid coordinates with background-position for page placement. I spent some time overhauling a site which had over 60 images (the vast majority less than 100 bytes). By the time I was through, there were 7 images and site load times were improved by almost 33%. Watch out for single pixel images which are used in repeat-x or repeat-y styles. You will have to create 2 separate sprites (i.e. repeat-x_sprite.png and repeat-y_sprite.png) to hold these 1-pixel instances in order to be able to correctly use the CSS sprite technique.

CSS sprites are indeed very cool. The heavy lifting is already entirely automated! The folks over at Website Performance host this great tool which takes a zipped archive of all the images on your page (up to 500KB) and actually generates a single image file with the corresponding CSS coordinate map. As you go through your original CSS file meticulously updating the relative px and em positions, you'll find the emcalc tool indispensable.

Multiple Domains

Most modern browsers invoke parallel, concurrent requests to separate domains. This means if you have your website hosted at 'www.mysite.net' you could host your images under images.mysite.net and the browser would invoke simultaneous requests (and process parallel responses) for javascript, css and image files. Website Performance Optimization has a nice write up of this technique where they conclude that up to a 40% improvement in web page latency can be attained in many sites.

But be careful not to overdo it - the sweet spot seems to be between 2 and 4 domains. Because of DNS lookups, you don't want to delay your page load in case of some troublesome roundtrip times. For instance, if the user gets unlucky 4x120ms ~= 1/2 second of blocked time lost (where no other content is loaded or rendered).

Caching AJAX requests

We all know the phenomenon of Web 2.0 and its AJAX underpinnings. What was such a boon to users and usability has pushed alot of poor web servers way over their thresholds. Consider a simple AJAX chat client. How often should you poll the server to see if a message has been written (or a friend is online)? Such seemingly harmless questions can quickly overload and kill a website. Let's say you have 100 chat users logged in to your site. You've decided to poll for new messages every 2 seconds and contact list updates every 10 seconds. In just one minute of elapsed time you're users have unknowingly generated over 3000 clickless requests.

If your server utilizes future Expire or Cache-Control headers for AJAX, and there hasn't been any update in message or contact list status, these responses can be pulled from the browser's local cache negating a round trip and greatly improving rendering speeds.

Loading of Components

Only load what you need when you need it. Sounds pretty straightforward but can cause some headaches to the unwary. Relevant stylesheets should be placed in the document HEAD using LINK tags to help make pages appear to be loading faster. Because the browser will start rendering these as soon as it has the data, items begin to popup for the user who's brain begins to interact with your site (load times feel faster).

Javascript should be placed as far to the bottom of the page as possible. This is because browsers stop all other requests and page rendering while parsing any scripts. While you may not be able to place all javascripts and their references at the end of the page, every one you do saves precious rendering time.

Avoid redirects and 404s

Redirects are often helpful for developers and sysadmins, but consume unnecessary time for the user. By default, neither 301 nor 302 statuses are cached by the browser unless Expires or Cache-Control are specifically added. A particularly wasteful redirect is when a trailing slash (/) is missing from a URL that should otherwise have one. For example, going to 'http://www.webmd.com/a-to-z-guides' results in two 301 responses; first to 'http://www.webmd.com/a-to-z-guides/default.htm' and then to 'http://www.webmd.com/a-to-z-guides/common-topics/default.htm'! If you're using Apache, you can fix this by setting up an Alias or mod_rewrite, or the DirectorySlash directive (for Apache handlers).

HTTP requests are expensive enough without getting a useless response like "404 Not Found". Particularly obnoxious is when an external JavaScript is taken down without warning. Not only will your browser block everything else while it tries to retrieve it, but it may even try to parse the 404 response as javascript, hoping against hope to find something usable. Yuck!

I've given you just a taste of some ideas for streamlining your website's content to delight both your users and your resident sysadmin. Next week, I want to give you some pointers on how to reduce the overall size of your web page to further increase load times and take a bite out of your monthly bandwidth bill. Subscribe to our feed to get it served up piping fresh in your favorite news reader.

Review: Joyent Accelerator 2.1 beta

Matthias Marschall — Thu, 06 Nov 2008 19:37:15 +0000

As you might remember, I recently interviewed Kent about the new Joyent Accelerators. I've now had a chance to give the new Joyent Accelerator template (version 2.1.3) a spin. The current release is 2.1.4 including some updates, which I'll point out below. Joyent provided me with two Accelerators for one week to try 'em out. Being a guy who normally tweaks everything to his liking (I'm kind of a perfectionist), I didn't expect much from the pre-installed software. Usually, when I start using a box with pre-installed packages I delete most of what's there and then install and configure to my preferences.

This was my first surprise in testing out the Accelerator. It comes with tons of software pre-installed, but, instead of generating an urge to get rid of it as fast as possible, I found everything set up very much in a way I couldn't have done better. In fact, I have to admit that I learned one or two things I could use to tweak my own setup ;-)

In the following sections I want to share with you the details about what I tested and how I liked it.

Welcome Email And Securing Your Accelerator

My first contact with the Accelerator was the welcome email containing the IP address of the Accelerator as well as usernames and passwords for SSH and webmin access. Additionally, there were some links to introductory articles on the Joyent wiki. What I missed was a link to the article which describes changing default accelerator passwords. It will be in the final welcome email. Kent already notified me, that the descriptions there were clarified based on feedback they received.

Package Management

Joyent does not use a pre-packaged version of OpenSolaris, opting to instead run a Nevada build. According to Kent, they do this to be able to get the most stability for their clients. Earlier this year they decided to switch from Blastwave to pkgsrc, the NetBSD package management system. They even host their own pkgsrc repository with a bulk load of precompiled packages optimized for running on an Accelerator. As I'm used to the OpenSolaris Package Manager IPS, I tried to play around a little with pkgsrc. The first thing I ran into was that I confused pkgsrc's pkg_add command with the additionally installed pkgadd commmand from Blastwave. I tried to follow the instructions to install a new package using the wrong command and obviously the syntax did not fit. To avoid that pitfall for other users new to Solaris, it might be better to provide only pkgsrc by default and an easy, optional way of installing Blastwave support if you need it.

Rails Stack

The new Accelerator comes with a full Ruby on Rails stack pre-installed. It features the latest versions of Ruby (1.8.7p72) and Rails (2.1). Unfortunately, Ruby 1.8.7 is not yet DTrace enabled. I had a short look at Joyent's Ruby DTrace patches for Ruby 1.8.5 and 1.8.6 but my make knowledge is too rusty to port it to 1.8.7 within a reasonable timeframe :-(

One thing I liked was that the accelerators have Phusion Passenger (aka mod_rails) pre-installed. I wanted to try that one out for a long time and now I had the chance. It was not enabled by default in the beta version I tested (2.1.3), but now, in the current 2.1.4 release, it is. Now starting a rails app is as trivial as typing rails myapp within your web root directory. I didn't do a heavy load test but it seems to work quite nicely.

Installing Wordpress

As I didn't find an immediate way to get wordpress from pkgsrc, I used the Subversion install method to get it running. Everything required by Wordpress is already pre-installed and set up. Subversion, Apache, PHP and MySQL are all there and waiting for you to give them some work to do. Using scot hacker's excellent script it took me less than 5 minutes to get Wordpress up and running on the Accelerator. I was impressed. Trying to do the same on a plain slicehost VPS took me 1-2 hours as I had to setup all the prerequisites in addition to Wordpress itself.

Summary

Getting started with a Rails project or a blog using Wordpress is a piece of cake using a Joyent Accelerator. I tried shared hosts like dreamhost and virtual root servers like slicehost, but never found the perfect combination of ease of use, power and flexibility. Personally, I really like OpenSolaris, and am currently switching our production environment from Debian + Xen to OpenSolaris and zones. ZFS, SMF and beadm already sold me on Solaris and I haven't even had a closer look at DTrace yet. To ease the (really nasty) pain of switching from Linux to Solaris, Joyent goes the extra mile to make you comfortable. Things like sudo and top, which aren't there in Solaris by default (approximated by pfexec and prstat respectively) are pre-installed. They might be the worse alternative in comparison to the native Solaris tools, but if you come from Linux, you will be glad to have them around 'til you get used to the native ones.

Website Performance Optimization

Dan Ackerson — Sun, 02 Nov 2008 06:52:00 +0000

Since Steve Souders's published "High Performance Websites" back in September 2007, a lot of people finally sat up and took notice of front-end web performance optimization techniques. Over a year later, you'd hope that most folks had figured out the basics like compressing content, using expires headers and reducing requests. Unfortunately, it seems that multi-core processors and bandwidth explosions have pretty much drowned out the awareness of this topic. Case in point, take a look at the search trends for mod_gzip vs. mod_deflate over the last 5 years.

Doesn't exactly represent a growing trend, does it?

Website performance optimization is yet another perfect bridge for developers and operations staff to come together and help each other out. Developers want their pages to load as fast as possible, delivering the best user experience. Sysadmins want to reduce load on their servers and decrease operation costs. To start out, have a look at some of the available tools you can use to measure the performance of your website.

Local tools for measuring

Firebug web development evolved
What started out as a javascript debugger for the Firefox web browser has evolved into the Swiss Army Knife of web development tools. The "Net" tab (pictured below) is what we're interested in : it shows the queuing of all HTTP requests and indicates which of these are loaded in parallel and which are blocked.

YSlow for Firebug
A Yahoo! development add-on for the Firebug plug-in (yes, you heard me right - an add-on for a plug-in!), YSlow actually grades your website against best practice optimization ideas.

HttpWatch
Developing for Windows and Internet Explorer? That's still not an excuse as the basic tool from Simtec Limited provides a comparable functionality to Firebug including the ability to decrypt and display HTTPS traffic.

Websites for measuring

Pingdom Tools - Full Page Test
An online (read no local network side-effects, but your site's performance measured straight from the backbone) HTTP request analyzer in the same vein as the "Net" tab from Firebug above.

Site-Perf.com Site Performance
Another HTTP request analyzer but with an added feature to choose one of three hub cities including New York & Amsterdam - very helpful for us folks here across the pond.

Web Page Analyzer Free Website Performance Tool and Web Page Speed Analysis
WebSiteOptimization.com's tool is a little less flashy than the preceding sites, but mixes in-depth analysis of HTTP requests with some best practice recommendations.

Interestingly enough, in making this post I realized that all these screenshots were becoming a bit expensive (5 images at around 320KB). So, I went back through and exchanged those expensive PNGs for comparable JPGs saving almost 50% of the total page size! Such simple considerations can mean the difference between your site surviving that initial Digg surge or conceding "502 Service Temporarily Overloaded" defeat.

That should keep you busy for awhile. Next week, I'd like to share some of the optimization experiences and techniques recommended by Yahoo's Exceptional Performance team. Subscribe here to add it to your favorite newsreader!

Kent Langley Talks About Joyent, Their New Accelerators And Their Use of Opensolaris

Matthias Marschall — Thu, 30 Oct 2008 19:28:13 +0000

Kent Langley is a Senior Director at Joyent, a cloud computing provider running and offering OpenSolaris Nevada based virtual servers, called Accelerators. Not only was he kind enough to answer my questions about his role at Joyent, details about the brand new Accelerator release, and Solaris, but he even brought in Ben Rockwood to give us an insight into the OpenSolaris community and Joyent's reasons for using pkgsrc.

Kent, you joined Joyent a couple of months back. What are your responsibilities as Sr. Director?

My full title is Sr. Director of Scale Consulting. My focus is a blend of business and technical goals. At this time I work in and manage the Joyent Channel Partner programs, Joyent Labs, and Scale Consulting Services. But, one of the most important things I do here is leverage my experiences to help customers large and small to understand and meet the challenges faced when migrating to a Cloud Computing infrastructure. One of the great things here is that migrating customers to Joyent is relatively easy overall because of how much the overall technology stack mirrors what they are often already used to seeing in their own data centers.

Joyent is releasing a brand new version of its Accelerator. What are the major improvements for power users?

Yes 2.1.x will be releasing in just a few days. The amazing teams here at Joyent created the Accelerator and all the surrounding technology that enables it to do what it does so well. It is an embodiment of a huge amount of Systems, Software, and Internet knowledge.

The biggest improvements for most users out of the box will be things like integration of Phusion Passenger (mod_rails), updated versions of Java, PHP, Ruby on Rails, Python, Erlang, Perl, MySQL, Postgres. We actually, for the first time, will be including ffmpeg and mencoder as well which is interesting if you need them. Also, all the celebs will be at the 2.1.x release party. Apache, memcached, varnish on Solaris, Merb, MySQL, Postfix, Mono, Subversion, Git, and a whole lot more.

Why did you switch from Blastwave to pkgsrc. And why did you completely remove IPS?

I'll roughly quote Ben Rockwood here. He says we did it for "fine grained control over versions of the packages that we need to support and use for ourselves and our clients. Also, Blastwave seems to have some stability issues."

We didn't really remove IPS. It's just not part of Nevada. So, it wasn't there to remove in the first place.

The switch from blastwave to pkgsrc was official in March 2008. One of the things that we have been working hard to do is make things generally easier for people developing and deploying internet applications in the cloud that want the power of Solaris, Dtrace, and ZFS but might be more familiar with other operating systems. Using pkgsrc allows us to take a step in this direction. Additionally, we maintain a significant pkgsrc repository for use by our clients. Recently a beta client said, "The new template is so fully configured that pretty much everything that had been causing me issues is now just installed by default." This is probably true for anyone that uses or ever used Solaris for anything. It's kind of a more friendly Solaris. An accelerator is a full featured building block for creating much more complex services while reducing operational efforts and TCO substantially.

You're currently running snv_89. OpenSolaris 2008.05 was snv_86 and current (as of this writing) is snv_99. How do you decide which build is stable enough for you to deploy? How much different is your version of OpenSolaris from the official releases? What do you tweak (besides using pkgsrc instead of IPS)?

Technically we don't use OpenSolaris. The naming conventions are unfortunate. The best way to think of it is that we do not use Indiana. We run the Nevada build. Again, to quote Ben Rockwood, "we do extensive testing of each release and determine which is best and most importantly stable." Stability is a primary concern, not bells and whistles.

You aren't using zfs as root filesystem for your Accelerator zones. Why not? Are there any serious technical issues holding you back or is it more a matter of taste?

Actually, we do use ZFS for all the filesystems here. It's just that most of the details of that are hidden at the Accelerator level.

You offer shared (Managed) Big IP f5 load balancers to be used to, well, load balance your customers applications. What type of access and configuration possibilities do customers have themselves? When would you recommend starting to use Big IPs?

Direct access to the BigIP's by customers is not allowed. It's essentially a managed service. We will work with clients to create and install custom IRules, VIP's, and any variety of setups. This is an extremely powerful part of our overall infrastructure.

A short list of the times to use the bigIP's is when your traffic patterns have either exceeded the abilities of a single Acclerator at the application or even the database level in some cases, when you want to use IRules to do application switching based functional partitioning, when you want to “hide” the real IP address of your machine from the world, to replace a software based load balancer that is no longer able to keep up with demand, or if you want to build in some High Availability. IRules are extremely powerful.

You send out passwords via e-mail. You ask your users to secure their Accelerator in your welcome mail, but there are quite some steps to be done. I know of hosting companies sending out passwords to mobile phones only. What are the reasons for your current approach. Do you think about changing it? Couldn't you e.g. offer (at least optionally) the possibility to upload a public key while ordering a new Accelerator?

Once your accelerator is delivered, it's security becomes your responsibility at the password level. If you change those passwords we can't even get into your Accelerators without some significant effort.

I think the concept of adding public key at order time would be nice. I'll float that to the team.

You are running all possible kinds of web application stacks on your Accelerators. E.g. you talked about a huge drupal site on your blog and joyent is known to host high traffic ruby sites. What are your personal preferences and why?

Yes, the Accelerator is a very flexible building block. A short list of some stacks would include Tomcat, Gigaspaces XAP, Axiom Stack, GlassFish, CakePHP, Symfony, Drupal, Django, Merb, Ruby on Rails, Mono, CouchDB, Ejabberd, and SunGridEngine.

We host some of the largest Ruby on Rails, PHP, and other sites on the internet using Joyent Accelerators. There are multiple sites that serve billions of page views per month. These are no longer uncommon. Around here I joke that a billion (pageviews) is the new million.

I am pleased that our overall technology stack works well enough to do this even with existing frameworks. But, it's rather brute force in many cases. For my personal preferences, I find most technology stacks and frameworks lacking for today's needs when peak loads can be 10x or more above the mean.

Cloud Computing, the actual act of doing computing in the cloud, has very strong parallels (no pun intended) to distributed programming models. But, there are only a fraction of existing web frameworks that do this well or even at all. The more interesting ones allow or provide fully distributed access to a horizontally scalable data layer either in memory or on disk. It's this data layer that's changing things because it allows a path to scalability that's been unavailable before and all to uncommon still today in every day use.

So, my personal preferences lean toward languages, frameworks, and products that follow the number of tenants - recently well put in an article in InfoQ about Ebay's architectural policies.

http://www.infoq.com/articles/ebay-scalability-best-practices

There are claims that the OpenSolaris community is heavily dominated by Sun. What are your experiences with the OpenSolaris community?

I leaned on our resident Sun Community expert, Ben Rockwood, again for this answer:

The OpenSolaris Community is heavily dominated by Sun. The OpenSolaris Community took an existing close-sourced OS with an enormous corporate infrastructure comprised of several thousand people and opened it to all. It is natural that the community is dominated by Sun given that those people have been working on the code for the last 20 years are from Sun. However, a large amount of non-Sun community involvement exists. There are well over 100,000 registered community members (more than 3 times the size of Sun's employee base) and growing steadily. Several non-Sun community members have served on the OpenSolaris Governing Board, including Ben Rockwood of Joyent. The CDDL was reviewed by Sun and non-Sun community members alike prior to its ISO launch, pre-OpenSolaris. Given that Solaris is the cornerstone of Sun Microsystems product it's likely that the community will always be dominated by Sun, but its important to distinguish between "comprising the bulk of..." from "iron fisted control".

What is your personal favorite feature in Solaris? Would you recommend using it? And why?

Broadly speaking there are two high profile features that make Solaris shine. One is DTrace and the other is ZFS. Once you begin to understand the power and flexibility of real time active tracing using DTrace it's hard to make do with other tools.

I would totally recommend working with Solaris. However, I will be the first to admit that there is a learning curve. It's not Linux, It's not BSD. It's Solaris and that comes in various flavors. As noted, our flavor of Solaris, the Accelerator, is different than most and more than likely a more gentle introduction without losing any of the great features.

Thanks a lot for taking the time to answer these questions.

My Pleasure. Thank you for being one of our best beta testers this round!

Visible Ops: Continuous Improvement

Dan Ackerson — Sun, 26 Oct 2008 06:47:00 +0000

Photo by furryscaly

In my last posts, I've introduced you to the IT Process Institute's "Visible Ops Handbook" which presents IT managers and system administrators with a clear road map of how to adopt best practices within their organizations and how to work smarter instead of harder. The first three phases outlined include change management, configuration management, and build processes. Today, I want to share with you their keystone practice that really ties all these pieces together in a single whole that will give your company the competitive edge in today's agile business world.

Metrics

You've been given the basic tools for this already. Change management is tremendously advantageous in the release process; without it your success rate for releases will be drastically lower and you'll undoubtedly have more unplanned work ("firefighting") in production. Configuration management dictates the quality of your builds and software library which will directly impact the resolution process (remember, rebuilding always beats troubleshooting!). The final phase discussed in "Visible Ops" is continuous improvement and it's basically about measuring and acting upon critical metrics gathered from your newly implemented change and configuration management processes. You cannot manage what you cannot measure, and I'm not talking about grepping through a log file here! You need to methodically plan which metrics will give you the information you need to improve upon your processes and ultimately deliver more business value per server than your competitors!

So what exactly should you measure? Again, the folks at the IT Process Institute give you a great head start. How effectively do you:

Release: generate and provision infrastructure?
Controls: make good change decisions that keep production infrastructure available, predictable and secure?
Resolution: diagnose and resolve issues when things go wrong?

Carefully analyzing and presenting these metrics should easily give you the mandate to:

increase the staff working on process areas early in the IT web operations life cycle (moving key staff into pre-production engineering roles)

increase time spent on proactive and planned work, instead of reactive and unplanned firefighting

increase the number of successful, meaningful releases

decrease variance between configurations, planned and actual work and even builds

Release Metrics

Time to provision known good builds: how long does it take to build and provision infrastructure from bare-metal?
Number of turns to a known good build: how many times must the build be modified before its acceptable for deployment
Shelf life of builds: how long will the build be in production until its replaced?
Percent of systems that match known good builds: how many production systems can claim this?
Percent of builds that have security sign-off: how serious is your organization about security?
Number of fast-tracked builds: how many builds were rushed into production via the emergency change process?
Ratio of release engineers to sysadmins: are you too busy doing stuff instead of thinking about it first?

Control Metrics

# of actual changes per week : how many were authorized?
# successful changes: what's the ratio of "emergencies" to "special" to "business as usual"?
# of service-affecting outages
# of hours spent on change management
# of changes submitted vs. changes actually reviewed

Resolution Metrics

MTTR (Mean Time To Repair): average time to restore service after any interruptions
MTBF (Mean Time Between Failures): the average time between service incidents

These numbers alone, however, won't convince your boss to turn the company upside down. You'll have to convince him through meaningful presentations demonstrating a clear business value in your revolutionary ideas. If this aspect of "visible operations" gives you any pause, then have a look at Matthias's gentle guide.

I've certainly given you plenty of food for thought this past month and really can't give enough praise to this 100 page volume of operational wisdom. If you're looking for a way to get your company out of IT chaos, "The Visible Ops Handbook" will start you down the right path!

Critical Bugs: Blame Or Chance?

Matthias Marschall — Wed, 22 Oct 2008 20:03:18 +0000

Photo by catatronic

Today we noticed that a feature we released yesterday created a really nasty side effect. While experimenting with a new content structure for AUDI, which was the only one we wanted the search engine bots to crawl, we accidentally invited the bots to crawl all content for all brands. As the content of all other brands was not migrated to the new content structure this bore the risk of spoiling our search engine ranking of our most important pages and thereby hurting our bottom line. Ouch. It was time for an emergency fix to stop the bots from reading too much stuff which might confuse their judgement about our site.

Root Cause Analysis

It's bad enough that such things happen once. But what I want to avoid by any means is that such a thing might happen again. Just fixing it and then going on as usual doesn't cut it for me. I really want to understand all the factors which led to such an error. It's not about blaming anyone but it is about learning what happened, and, even more important, what to change to prevent it from happening again.

So, which forces were at play? Our usual process for critical changes like yesterday's looks like this:

The Business Owner enters a user story into our backlog. Usually the story describes how a certain feature shall look.
The developer starts implementing the story in a test driven manner
As soon as the story is ready for test, the Business Owner tests it and signs it off
If it's critical enough, I usually do some more rigorous testing for side effects
Then the story goes live

Usually, this process ensures that we avoid releasing badly broken stuff. We have enough quality gates in the way to catch nearly all the bad guys. So how could this one sneak through? Thinking about it for a minute, I immediately came up with a list of stuff to further investigate:

Was there too much logic in the views? Is the stuff really unit tested?
We already developed the same story once before and dropped the result as it didn't suffice. So this was the second approach to the same issue. It might be the case that we didn't judge the side-effects as serious as the first time (wear-off effect)
Only "positive" testing. Sign off testing focuses quite mainly on "Is the new stuff there?". I have the feeling that we don't do enough "negative" testing like "Is this old feature broken now because of the new stuff?"
As I was busy with other tasks, and we already tested it once, I didn't give it a second look

Looking at those points, two root causes seem to surface: Lack of well defined, acceptance test criteria and lack of peer review.

Writing Bullet Proof Acceptance Test Criteria

While fleshing out a story, we do a very detailed concept of the desired effects: How shall a new page look like?, Shall the new links be 'follow'?, etc. Sometimes we mention some invariants like: "Keep all other pages like they are" or "This does not apply for that page". But what we usually leave out is describing test cases to verify that everything works as expected:

Given the brand AUDI, the links in the "top-link-box" shall be follow
Given any other brand, the links in the "top-link-box" shall be nofollow
...

So far it worked out, but this time we got bitten :-(

Good acceptance test criteria should guide a tester to validate all critical points of a new feature. The most difficult part here is to find out which side-effects might come up and how to test for them. Finding those usually requires a detailed understanding of the dependencies within the application. Those are usually best understood by the developers. Because of that, it's usually a good idea that the developer adds some test criteria as well. He knows better than anyone else what might break.

Peer Review Saves Your Day

As we're a very small team where each of us has comparable experience, we trust each other completely. This is a good thing, of course. But, it leads to the negative side effect that we don't do enough code reviews. In my experience, everyone can only profit from a code review: The reviewer practices his code reading and feedback giving skills, and the coder being reviewed gets to practice explaining his code and perhaps some additional insight into any trouble spots.

The same is true with testing. Too often we, the tech folks, think: Business signs it off, so it's good to go, right? But, we should know better. More than once, I've double checked a signed off story and found bugs in it. So even for testing, peer review is a helpful thing. It should also help in improving your test cases (see above).

Actions To Improve The Status Quo

Neither of these root causes can be quickly fixed. Both require changing habits rather than just flipping a switch. What might help improve the situation with acceptance test criteria is adding a template for User Stories which explicitly asks for this test criteria. It could look like this:

Steps to verify this story:
*
*
*

Steps to verify nothing related is broken:
*
*
*

Cucumber, a tool to make feature documentation executable, might be one step further in that direction.

Finding concrete measures for increasing our frequency of code reviews is a little harder and will need some team discussions. Let's see what we come up with.

Conclusion

Our tightly knit team is doing a great job already. Still, if a critical bug sneaks through, I want to use it as a chance to further improve our processes and as a reminder to remain vigilant. Continuous improvement of processes is as important as keeping your code clean!

Presentations Without PowerPoint - Tools For Bridging The Gap

Matthias Marschall — Thu, 16 Oct 2008 20:12:30 +0000

Last week, I introduced three key factors to help you connect to a non-technical audience. This week I want to introduce you to some tools and techniques for preparing your presentations. And, no, I'm not going to talk much about using PowerPoint ;-) ...

Know Your Goal

Every good presentation starts with a message. What do you want your audience to remember or do after your presentation? Sometimes it's as easy as "I want approval to buy that new server" but sometimes it's not as clear. Teaching an Agile class is a good example. What single thing do you want your students to remember? Agile processes are such a wide field, it's hard to pick your goal for the class. Even if it sounds pretty obvious once you read it, I had to rule out lots of possibilities before coming up with "Be able to conduct a software development project using agile practices" for the course I'll be teaching again this November. It's not "know all the various agile approaches like Scrum, XP, Lean, etc" or something similar. It clearly focuses on being able to use Agile, not just know about it. That set the direction for how to present Agile to my students.

Structure Your Thoughts

Once you know what you want it's time to find a way to get there. I prefer mind mapping in the early stages of structuring my thoughts. Even though it is sometimes preferable to leave your computer and do your brain storming offline, I regularly use mind mapping tools to capture my thoughts. Using software in the process simplifies organizing and sharing your ideas.

If you want to use a tool I recommend freemind. It's Java based , open source and really gets out of your way when you use it.

Recently, I started playing around with Mindmeister, a web based alternative. It imports freemind files and offers the advantage of easily sharing and collaborating on your mind maps. There is a free, entry-level version, but, if you need it, the full-blown product will cost some money.

Create Graphs And Diagrams

If you want to visualize a complex process or structure, you'll need to draw a diagram. While experimenting with a lot of tools over the years (like MS Visio, StarDraw, etc), I never found one I liked. I'm looking for something plain and simple, but most tools I tried out were bloated with features.

Not so long ago I came across gliffy.com, a flash based app which does just what I need: Draw a simple diagram no matter on which platform I work. You can create public diagrams for free. For serious business use you'll have to pay $5/month for an individual license which gives you private and ad free diagrams.

Find Creative Commons Licensed Pictures To Support Your Point

If you want to enrich your presentation with pictures to better connect with your audience, you might wonder where to get them from. Luckily, flickr provides a specialized search for pictures published using the creative commons license. Sort the results by "Most Interesting" and you'll get really good pictures.

If money is not too much of an issue, sites like iStockphoto provide you with perfect pictures for any kind of presentations. For a presentation, small images should be sufficient, which set you back $3 each.

Not So Mainstream Presentation Tools For PowerPoint Haters ;-)

Finally, after defining your goal, structuring your message, creating some diagrams, and finding photos supporting your message it is time to come up with some slides. Sure, you could fire up PowerPoint, Numbers, or OpenOffice, but hey, you're a geek, right? If you prefer to avoid the "uncool" stuff ;-) have a look at these alternatives.

LaTeX
There are at least two serious ways for doing presentations in LaTeX. If you are already familiar with LaTeX, these might be for you: Prosper or Beamer. I used LaTeX to present my diploma thesis and was very glad about the possibility to use a normal text editor without all the bells and whistles which usually come with more graphically oriented presentation packages.

Java
If you are a real code slinger, you might even want to code your presentations. While this approach is bound to Windows as a platform, it is definitely the coolest way of producing PowerPoint presentations ;-)

Just download the Apache POI project and just code a new slide show:

    //create a new empty slide show
    SlideShow ppt = new SlideShow();

    //add first slide
    Slide s1 = ppt.createSlide();

    //save changes in a file
    FileOutputStream out = new FileOutputStream("slideshow.ppt");
    ppt.write(out);
    out.close();

Adding new pictures is easy, too:

  // add a new picture to this slideshow and insert it in a  new slide
  int idx = ppt.addPicture(new File("clock.jpg"), Picture.JPEG);

  Picture pict = new Picture(idx);

  //set image position in the slide
  pict.setAnchor(new java.awt.Rectangle(100, 100, 300, 200));

  Slide slide = ppt.createSlide();
  slide.addShape(pict);

And if you really keep your slides simple, this is about all you need to do. For a complete list of code examples check out the Busy Developers' Guide to HSLF drawing layer.

By the way: There is even a Ruby binding for Apache POI ;-)

SlideRocket

You aren't convinced that coding your presentation is the way to go? Hmm, but you still want to be cool, right? Why not try SlideRocket, a flash based PowerPoint replacement with some really nice features like sharing slides, doing online presentations and directly searching for pictures from within the application. It even has an offline client based on Adobe AIR.

Currently, SlideRocket is my preferred way of creating presentations. What are your choices? Do you have good reasons to stick with PowerPoint? Or are you more the "Source Code Guy"? Share your approach and favorite tools in the comments.

Visible Ops: If At First You Don't Succeed, Build, Build Again

Dan Ackerson — Sun, 12 Oct 2008 06:15:00 +0000

Photo by benchun

By now you should have a better understanding of how your data center is setup and exactly what's all in there. You've been carefully monitoring changes to this environment and are ready to take it to the next level. The "Visible Ops Handbook" introduces the concept of "production fuses" : when things go wrong with a fuse, you don't try to repair it; instead, you simply replace it with a working one right out of the package. Not only does this require less downtime, but you can be pretty sure that the new fuse will work as expected (and not accidentally burn down your entire home!). Your confidence in the correct functioning of the new fuse is directly related to how well you've been monitoring changes on that production fuse. If some engineer has made undocumented fixes or "improvements" to it (which is probably what caused it to short out to begin with), then the pre-fabricated fuse you replace it may cause unintentional side effects and further headaches. Replacing instead of fixing is what all high-performing organizations do, giving them high server to sysadmin ratios, less unplanned work, and the ability to maintain manageable system configurations.

Create A Release Management Team

This team's primary responsibility is constructing the mechanisms to deploy the best configurations into production. They won't be doing the builds; instead, they'll be designing the build. You'll want your best and brightest for this critical task. Take the most senior IT operations folks and get them out of reactive, firefighting and into a more proactive, structured regimen of release engineering designed to decrease downtime and increase your successful change rate. To reduce complexity and cost while improving the manageability of your operations environment, note the following benefits of rebuilding infrastructure as opposed to repairing (in increasing order of importance):

An automated process takes a known amount of time, as opposed to firefighting which almost always takes longer than original estimates
It introduces less configuration variance, as opposed to repeated break & fix cycles ("What all did we just change to get it to work now?")
They are usually better documented and less complicated so they can be monitored by junior staff which frees up senior staff from firefighting
When senior staff break free of firefighting, they can work on new build projects fixing other systemic issues (addressing the real root causes of issues instead of just their symptoms)

Create A Repeatable Build Process

From the "Find Fragile Artifacts" step I introduced last week, we have our starting point in this next project - namely, getting fragile infrastructure replaced with stable builds that can be run by junior staff. Infrastructure blueprints should be stored in an online, protected, backed-up repository known as a Definitive Software Library (DSL). This library not only houses all the relevant software applications your environment requires to operate, but also license keys, patches, etc. The release management team should now consider the following:

Identify the common set of essential services and components used across your infrastructure (OSs, applications, business rules, and data)
Create a list of standardized components, called a "build catalog." Look for ways to create components that can be reused and combined to create standardized configurations (i.e. Apache and Oracle both installed on a Solaris OS)
For each component in the build catalog, create a repeatable build process that generates it. The goal is to have an automated build system that can provision the package by "pushing a button" (i.e. NIM for AIx or Jumpstart for Solaris)
Any testing environment should be isolated from the production network to ensure that it does not disrupt production systems and to make sure that all dependencies outside the test environment are fully documented and understood
Ensure that you can recreate each system with a bare metal build. Our goal is a repeatable process that eliminates anything tedious and error-prone, as well as reducing work and errors (i.e. virtual images from Xen or VMware)
For critical HA or load-balanced environments develop reference builds that can provision a box from bare metal without human intervention (i.e. triggered as a workflow action based on a certain Nagios alert)
When the build engineering process has been completed, store them in the DSL

Create And Maintain The Definitive Software Library (DSL)

Let's face it - it can be downright horrifying having to check out some source code from a development repository for release at 1am in the morning. As a sysadmin, you cringe while executing the build script, cross your fingers and plead "Will it build without any errors?" If you weren't able to convince the development group to adopt a Committer Role, here's your second chance. Let the release management guys build it for you and commit this into the DSL (and this at 1pm in the afternoon when they can easily go track that developer down who made that "oh, just one more quick fix" commit) for the operations team to confidently release.

Designate a manager to maintain the DSL who will be responsible for authorizing the acceptance of new applications and packages
Create an approval process for accepting items into the DSL
Establish a disparate network to house the DSL and the required build servers (i.e. a DMZ)
Any software accepted into the DSL (both retail and internally developed applications) must be under revision control
Audit the DSL to ensure that it contains only authorized components

Close The Loop Between Production And Pre-Production

So you have a beautiful test environment upon which you're running production infrastructure - or so you think. Do you have automated detection of changes on all production servers? Are you sure that not a single developer has access to production? Even if only your operations staff has access, as most systems nowadays number thousands of files and perhaps hundreds of separate configurations, the odds are against them for remembering to document every single production change. You have to use a some automated configuration management tools (like Cfengine or Puppet) to ensure that production builds stay in sync with the golden builds in your DSL.

If you've made it to here, you should have reduced the amount of unplanned work down to 15% or less. The ITIL Institute reports that by drastically reducing configuration counts, you can significantly change the staffing allocation from unplanned to planned work, and consequently increase the server to sysadmin ratio. Now you've:

created a release engineering team to define and generate infrastructure that can be repeatedly built

further decreased uncontrolled production changes, which increases the amount of time available to work on planned tasks

created a new problem resolution mechanism, making it cheaper to rebuild than to repair - a viable alternative to protracted fire-fighting

enabled shifting of senior staff from the front-lines to the release management area, where the defect repair costs are lowest

closed the loop between production and release management, to curb production configuration variance

enabled the continual reduction of unique configurations in deployment, increasing the server to sysadmin ratio

Stay tuned next week for some concluding words about the Visible Ops Handbook and how to pull it all together!