Cloning and YouTube.com

A couple of months ago we started a new project. I took a noble position of Software Architect and a small team of developers to create a big and complex solution - although it's not quite true but let's name it a clone of YouTube and FlickR at the same time :). Yes, I know - the internet is full of them and constructing just another one should be a boring thing... But for me it was an excellent possibility! Why? Let me explain...

After the tremendous business success of first web2.0 applications a lot of customers of our company became ordering their clones. It was a slow process with a lot of hesitation and afterthought. After Google acquired www.youtube.com every internet business owner start having nightmares. All of a sudden everyone understood how incredibly profitable their business could be even without any visible business model if they had been more active. As a result everybody rushed forward but unfortunately with the same level of stupidity - the vast majority decided to catch the same train. Yes, they start dreaming about YouTube clones! It is not hard to construct something similar to YouTube (and even easier to order its construction to outsourcing company like one I work for - you don't even need to get your hands dirty managing developers or coding directly). The task could be easily fulfilled by a team of four web developers in two-three months period (plus a tester to make the application stable). So if it is so reachable and cheap why not to try? And they tried and tried a lot... The only problem here is that this estimation remains true as long as the load of a site is small - in other words when you have hundreds (not thousands) visitors per day and they don't upload much stuff. But this was sufficient. In spite of the fact of willing to succeed almost all customers we had until now wanted to bite off just a tiny piece of YouTube market. Truly speaking they merely dreamed about becoming at most middle-sized internet company in the distant future. They preferred to be born small, hope to grow big and die huge (just kidding ... sure all we want to get on top and be there forever :) ). So we were making blind copies of the same site again and again but on a very small scale. We were copying the design, functionality, trying to use the same open source software as YouTube does and so on. It was a boring stupid work usually assigned to teams formed mostly from students as they performed work like this for the first time and didn't feel sick... Of course the quality of solutions were not high but well enough to work most of the time and made customers happy.

But finally one of the customers decided to be born huge at once. He came to a wise and simple conclusion - not to create dump clones of famous systems but provide some flexible instrumentation which helps in construction of media sites. Unfortunately I can't provide the full description of the business idea but it is not necessary. The only thing that worth mentioning is that due to the specific market orientation such none-trivial requirements as scalability, high-availability (99.999% up-time) and total visibility became the core features of the architecture.

Why did I write such a long story? Each technical solution has the appropriate business context. I'm not only planning to share the most difficult technical parts and possible solutions on the pages of this blog but also want to discuss their applicability to a concrete business areas...

Stay tuned!

Guy Kawasaki - The Art of The Start

When this man speaks everybody listens! This time Guy Kawasaki is talking to entrepreneurs about The Art of The Start - both the book and the strategy of achievement. If you haven't seen this video yet I highly recommend you to do so. It is smart and funny as everything I heard from Guy so far ...

As The Matter of Testing

Okay, a few words about testing now...

As I wrote before I'm working on a huge project involving a lot of technology and hardware:

1. database clusters,
2. web clusters,
3. computational clusters for concrete algorithms,
4. all mentioned above distributed across a few data centers,
5. mix of Linux and Windows platforms etc.
   

To my mind the project is not typical for an outsourcing company. As you might suspect outsourcing companies usually don't have resources and processes established for development of big systems. Also they don't bother much about quality (no matter what they tell you and which certificates they have) what is crucial in construction of a 24x7 solution. But this is the core of the business - they do not sell products, they sell people (more on this in separate posts). Nevertheless every company has a QA department. So do we. The only problem is that the QA staff is not technical in nature. In other words we don't have software engineers in testing at all. To be honest there are few projects which really need something more advanced than just clicking through the user interface to make sure it works and meet the specification. A small numbers of exceptional projects which needs sophisticated testing tend to have some developers assigned to this activity inside the project team. Sounds not bad - at least we have testing in place. The only problem is that there are a few questions each team has to answer first:

1. how to setup the testing process?
2. what and how to test?
3. how to accumulate the results?
4. what conclusions can be made on the basis of the data collected?

As you probably understood a team needs to create the whole testing process from scratch knowing literally nothing about how to do it efficiently. In our case the additional question arose: how to do it right for a 24x7 system?

Fortunately we are not alone on the planet :). Here is the excellent video from Google which answers some of the questions above. Enjoy!