New server for API

2008-05-07 23:51:52 | johann | #

After the Browsershots servers were overloaded for several weeks, I have finally added a new powerful machine for the FactoryInterface. It's a brand new Dell PowerEdge 2950 with two Xeon 5405 processors (8 cores total) and 8 GB of RAM, and I'm renting it from our hosting sponsor QuickHost.com. The subdomain api.browsershots.org has been switched to this new server, and factory admins don't need to change anything. This upgrade resolves the 503 Server overloaded message that has been so common recently.

The web and database server are still running on two dual-core virtual machines (using VMware). Since the primary web server doesn't have to handle screenshot uploads anymore, the response times for the web interface are also improving. However, the new bottleneck is now the database server. I'm planning to reduce database server load by introducing memcached in the next weeks.

Name server downtime

2008-03-02 07:21:58 | johann | #

Our local name servers went offline yesterday. I'm now using OpenDNS instead. The screenshot service was unavailable for 85 minutes, and some emails were delayed by up to 12 hours, but now everything should work again. Sorry for the inconvenience!

Using Amazon S3 for screenshot storage

2008-02-22 21:40:41 | johann | #

Browsershots is on the frontpage of del.icio.us again, and our servers are slightly overloaded. So I decided to move the screenshot storage to Amazon S3 (Simple Storage Service) to reduce the number of HTTP requests per second and improve the page load times. But the biggest advantage is that the space for screenshots is no longer limited by our 300 GB hard disk. So far, it seems to be working great.

Priority processing for your screenshots

2007-11-16 22:30:56 | johann | #

If you send me 10 Euros or 15 Dollars, I will give you a month of priority processing. This means that your URLs don't have to wait in the job queue, they will be processed before all other requests. You will usually get 10 to 20 screenshots within 1 to 3 minutes. This is similar to DomainPriority which some early adopters may remember, but now it's attached to your user account and includes all domains that you may want to test.

Update: I have integrated payment processing into Browsershots, so that your priority will be activated instantly. And I've added a new page with PayPal buttons for different currencies:

http://browsershots.org/priority/

I also have a bank account in Germany. If you want to transfer money directly, send email and ask for details. Financial support will accelerate the project development considerably. I am a full-time student and work on Browsershots in my spare time, but I have to take closed-source jobs to pay the rent. I also offer consulting services, e.g. if you need help setting up a Browsershots system for your company.

As a screenshot factory administrator, you'll get automatic priority processing if all your factories together have uploaded 1000 or more screenshots in the last 24 hours.

Release 0.4-beta2

2007-10-27 00:24:01 | johann | #

Today I have released the second beta of Browsershots 0.4. The source code was updated to work with the latest Django. Several language translations were added. The biggest new features are self-registration for new users and standalone mode for the screenshot factory. A detailed changelog can be found in milestone:0.4-beta2.

Source code downloads are now available from SourceForge:

Or you can get the latest source code with Subversion: 

svn checkout http://svn.browsershots.org/trunk/shotfactory/
svn checkout http://svn.browsershots.org/trunk/shotserver/

Thanks again to my hosting sponsor QuickHost.com for upgrading the database server's hard drive to 300 GB. At the moment, screenshots can be stored for a whole month after the last access.

Hard disk upgrade

2007-10-04 00:31:29 | johann | #

The storage for screenshots on the ShotServer has been upgraded from 100 to 300 GB today, thanks to our hosting sponsor Quickhost.com. Because of the growing popularity of Browsershots, screenshots had to be deleted 5 days after they were last viewed, but now we'll be able to keep them longer again. How long exactly remains to be seen, as this depends on the number of screenshot uploads per hour, and on the number of visits to old screenshots.

The unscheduled downtime last weekend and today was due to this hard disk upgrade and some networking problems on the virtual machines. Sorry for the inconvenience. I hope Browsershots will run smoothly for the next months.

Release 0.4-beta1

2007-08-21 01:08:58 | johann | #

Finally, the first release of Browsershots 0.4 is out. The server software has been rewritten completely, using the Django framework. Details can be found in milestone:0.4-beta1.

Source code downloads are now available from SourceForge?: http://sourceforge.net/project/showfiles.php?group_id=202792

The browsershots.org address was switched from version 0.3 to version 0.4 yesterday. The new server is sponsored and hosted by QuickHost.com. It runs on two separate virtual machines for database and web server, and I'm very happy with the performance so far.

Release 0.3.0

2007-08-19 08:34:27 | johann | #

This release concludes the development on the old version 0.3 of Browsershots. The development focus has shifted to version 0.4, and little has changed in the 0.3 branch since the last release. A quick overview is available in the milestone:0.3.0.

The old shotfactory and shotserver packages have been moved out of the trunk and replaced with version 0.4 today:

If you have a working copy of the ShotFactory for 0.4 checked out, you can adjust it like this: 

svn switch https://svn.browsershots.org/trunk/shotfactory

Designers wanted

2007-07-27 00:45:26 | johann | #

The new version 0.4 of Browsershots is online for testing. It's not feature-complete, but the core functionality is there. Now I'm looking for design ideas for it.

If you're a web designer and want to contribute, this is your chance to become famous! Your name will be listed on a contributors/credits page, but that page still needs to be created (and designed). Email me to get in touch.

Any of the following contributions would be hugely appreciated:

If there are several different good designs, I'll make a style switcher (similar to the language drop-down box). Your design will be published under the GNU GPL like the rest of the project.

(Question: Is this done? Is there a deadline?) :)

Is anyone working on the logo?

Need update on this... :-)

Sorry - i do not know how to give you this ;) new design

Release 0.3-beta1

2007-05-31 23:31:29 | johann | #

Another preview release. See the changelog in milestone:0.3-beta1 on the roadmap.

As before, the source code is licensed under the GNU GPL. It can be obtained directly from the Subversion Repository or in ZIP files from the download area. There's also a binary package for Windows that comes with all dependencies, including the Python 2.5 interpreter and all libraries.

Exploring Django

2007-03-02 23:37:17 | johann | #

I'm considering the Django framework for the next version (0.4) of Browsershots, and I have started a source code branch for that. Using Django gives us all of the following for free:

Some browsers unavailable

2007-02-26 10:40:37 | johann | #

I'm moving house right now, and most of my screenshot factories are stowed away for shipping. I hope to get them back online within 2 weeks. This means the throughput of Browsershots is currently very low, and not all platforms/browsers are available. If you want to help (and have a desktop or server computer with broadband internet), you can run a screenshot factory too. See HowToCreateNewShotFactory for details.

Firefox on Mac OS X

2007-02-08 00:29:35 | johann | #

Finally I have also added support for Firefox on Macintosh to the ShotFactory program. If you want to add Firefox on your machine, see HowToUpgradeShotFactory, then send me an email and tell me the User-Agent of your version of Firefox.

Firefox is not very scriptable with AppleScript?, but with the "System Events" application we can resize the window and send key events for scrolling. The source code uses appscript.

Firefox on Windows

2007-02-04 12:20:59 | johann | #

I have added support for Mozilla Firefox on Microsoft Windows to the ShotFactory program. If you want to add Firefox to your screenshot factory, just send me an email and tell me the User-Agent. See HowToUpgradeShotFactory for detailed instructions.

Also, the ShotFactory on Windows now attempts to empty the browser cache before each screenshot.

Internet Explorer is back

2006-12-23 00:28:17 | johann | #

My screenshot factories with Internet Explorer on Windows have been offline for most of the last month. After each restart, they would crash again quickly. Unfortunately, I could not investigate the problem directly because I'm still in Seattle.

Talking to my flatmates in Stuttgart on Skype, I have now finally found the problem: the hard drives were filled up with gigabytes of Temporary Internet Files in the browser cache. Old files in the cache didn't get deleted because the ShotFactory script kills Internet Explorer after each screenshot, without closing it properly.

So now we deleted the browser cache manually and restarted my Windows factories quad and sven with Internet Explorer 5.0, 5.5, 6.0 and 7.0. Hopefully they'll keep running until I return to Stuttgart in the beginning of January.

Visiting Seattle

2006-10-25 05:41:56 | johann | #

I'll be in Seattle for one month starting tomorrow. If you want to meet me for a cup of coffee, drop me a line.

Some of my screenshot factories in Stuttgart will probably crash from time to time, and it will take a little longer than usual to get them back online. Thanks for your patience.

Update 2006-11-26: I have extended my stay in Seattle for another month. I hope to return to Stuttgart in the beginning of January 2007.

Windows Binaries

2006-09-27 14:28:15 | johann | #

I have created a binary distribution of the ShotFactory program for Windows. It was built with py2exe and has the following benefits:

Get the experimental Windows binaries from the binaries download section and read the updated installation instructions: InstallFactoryWindows. Please try and report any problems that you may find. If you have never run a screenshot factory before, have a look at HowToCreateNewShotFactory.

Hard Disk Maintenance

2006-09-22 15:42:30 | johann | #

I found that one of the disks in the server's RAID-1 had read errors in the weekly selftest, so I decided to rebuild the RAID from the other drive. During the rebuild, I noticed that most screenshots were blank because the server was too slow when redirecting browsers. So I put one hour worth of screenshots back into the request queue. Then I shut down the web server until the RAID was finished rebuilding.

The result was roughly two hours of interrupted or slow service between 7am to 9am UTC today. Thanks for your patience. Some technical notes can be found in ShotServerRaid.

Release 0.3-alpha2

2006-09-12 21:45:49 | johann | #

A new technology preview has been released today, with a number of new features. A summary of changes can be found in milestone:0.3-alpha2 on the roadmap.

The source code is licensed under the GNU GPL. It can be obtained directly from the Subversion Repository or in ZIP files from the new download area.

Dugg Again!

2006-09-07 06:04:33 | johann | #

Hello and welcome everybody. Browsershots is on the frontpage of http://digg.com/ again. Sorry for bad server performance, but there's just too many of you. I increased the number of server processes (MaxClients) from 64 to 192, but the status page shows that the server is still overloaded. This is the first time that the new server gets some real traffic, so thank you for the testing.

And for all of you wondering about the length of the queue: the maximum wait is now 4 hours, and if you don't change it from the default it's only 30 minutes. After that, your screenshot requests simply expire. That means that some or all of your requests will be ignored, but the queue will always be 4 hours or less.

Update: There were also some errors because the database connection limit was exceeded. I increased that from 100 to 300. Sorry for the hiccups, and thanks for your feedback.

And here's some technical background info about this onslaught:

Note: See SimpleBlogPlugin for information about the blog view.