Our Companies databases are more complicated than twitter and facebook

Just got an idea to find out what are the databases used by the big heads in IT these days

Had a small search and found out the following

 

Google uses BigTable database(created by google)

Twitter uses MYSQL as database ! [I think they moved to some NOSQL kind of DBMS now]

Facebook uses MYSQL as database ! [I think they moved to some NOSQL kind of DBMS now]

( at least these two were using MYSQL the days the article I just read about them were writtern)

 

Well ok we understand the case for google it for sure needs a really extraordinary DBMS that can handle its unique requirements, but what about twitter and facebook!

These two giants have that huge data , and really fast growing database, with huge audience accessing and updating these database rapidly.

It really shows how much power an opensource DBMS like MYSQL can provide and again raises the question why many companies in our part of the world insist their DBMS must be either Oracle or Microsofts’ SQLSERVER !!! while 99% of the companies around here will never require more than the amount of data, number of users, powerfull security that MYSQL provides to Facebook and twitter.

 

What is the problem with the decisions of these companies ?! if you know the answer please let me know in the comment

 

24 comments

  1. Well.. There are many views and possibilities!
    For instance, facebook & twitter for sure have some GURUs who are able to maintain things very well. But on the other hand, we dont really have such people.. Or if we do, the higher management doesnt believe on that.. And prefere to have a full support by the providers..

    Just a quick passing :)

  2. thanks qais agreee to your comment , but still i think if really the paid versions of DBMSes like orace and SQLSERVER was better than the free MYQL then most probably facebook and twitter used it , why decision makers in the region don’t see his as fact and continue paying to oracle and microsoft, i really liked when your organization started giving attention to open source. hope it can help companys to know better about what they are paying for and what they can get better without paying for it

  3. You can say we like our breakfast on bed, thats the best metaphor I could come up to explain your scenario

  4. comment from G+
    Tariq Al-Suleimani – becouse they have alot of money and they dont know how to use it 

  5. And your bank? Do you think they will use NoSQL?

    Twitter/Facebook has very simple data structure. Consistency and reliability is not a strong issue (if a tweet is lost the more likely is that nobody will really care). They don’t make significant money so every gain they can have in datacenter is good.

    They use a very limited and specialized tools that are innexpensive to use and very effective in their limited scope.

    This doesn’t mean that typically every other software you’ll enconter is a social media like web site, with the same need and features.

  6. Good point Nicolas.

    but are you suggesting that MySql, and BIGTABLE have consistency and reliability issues comparing to paid DBs?

    and there are hardly any ground breaking features that the paid DBs have over the free good ones (e.g. MySQL).

    IMO, its purely mental thing, think about it, if you are clueless in technology, and someone comes and provides you with one free and one paid DB and tells you they do the same thing, you would think that (hmm, my business is important, so am sure the paid version is better, otherwise why is it “paid”? and beside, the “paid” DBs know how to market themselves). this will not be the case if a tech savvy handles the business, because he will know that the things that needs to be done can be done with the free DBs, so why cost more

  7. comment from G+
    Omaima Saleh – you know …. it could be that they just didn’t have someone to tell them and convince them that there are other options that could be better and cheaper than what they have
    Yesterday 11:20 AM

  8. +Tariq Al-Suleimani lool agree, but it is private sectore they should be finding ways to save money !
    +Saud Al-Zakwani may be that is correct, since they already hired people expert in these DBMSes its hard to change (but) if they changed they will save money on (resources ppl) on (DBMS licenses ) so still they should be looking into it

    +Omaima Saleh thats correct also add to it the huge Ads by Oracle and Mircosoft about their products IT decision makers fall to these things easily

  9. The two examples you mentioned (twitter and facebook) although use/used mysql – it isn’t in the traditional RDMS fashion. Lot’s of references are available on the web on their use – but two points stick out –
    – no referential integrity
    – no joins

    So their use would be more-so classed as a bare-bones data-store from which they build higher level functionality in the application layer to cater for their needs.

  10. I used to work for à big organisation, my job was to maintain and develop the ESB for the organisation, I have had an insight in many big and smaller db’s most of them Oracle or MsSql. Most applications using Oracle where good and proper apps, with a good db scheme, most application using MsSql where bad and ugly apps with some extremely bad db’s schemes (no normalisation or what soever). But for every application and its database they all could work perfectly with a MySql db, there was no need for an Oracle or MsSql db, also the ESB where I used to work with was Oracle, this is one of the worst apps Iever worked on (the only thing helping was praying). I made a simple, php xmlrpc based solution, it worked was many times faster then Oracle ESB and had no downtime, but it was made in php, used a MySql db and didn’t have a proper ui (I could make it but that would require time), the management, the dumbheads I called them, decided to stay with Orcale and have huge problems which could only be managed by expensive Oracle consultance. iIn the end I left and after 8 months they still have the same problems, pay a lot to maintain bad software with an expensive name on it.

  11. I think that’s an easy one: most big companies are rather old and therefore use `older’ techniques. Facebook sprung from a student mind, so he had little money to buy anything expensive: i.e. an open source DBMS where MySQL is the most popular. I guess it’s something similar for Twitter.

    Large companies want their support with the product and you can negotiate some kind of SLA when you need one (or you find a third party which lets you do that). Also don’t forget Oracle and MS have more budget for their PR, marketing, … to convince decision makers or the executive level, which might impose a certain product upon the devs.

    Also, it’s not always the amount of data that does it; but rather what you want to do with it. The last time I touched a DMBS from close range (few years back), there were some things the large vendors supported which was not supported from within MySQL.

  12. Interoperability. This is why we are forced to use oracle. Lots of 3rd party vendors only support their product if a major db is used. So many people in this trap.

  13. @alvin
    thank you very much for the information related to the twitter and facebook even if they are using mysql they are not using joins neither referential integrity i didn’t notice that.

    still i have not found any topic stating that MYSQL have performance problems in any of (joins, referential integrity)

    thats why i posted this that as oracle and MSSQL SERVER are RDBMSes MYSQL is also RDBMS but there is huge difference in pricing , why companies (that are usually looking to reduce expense and increase revinue) still pay lots of money to expensive RDBMSes

    other than Ads and marketing done by Oracle and Microsoft , i can’t find any technical reason makes a company decide to use Oracel or SQLSERVER instead of MYSQL

  14. @Irkan NoSQL database have “eventual consistencies”. NoSQL databases can’t perform advenced requests in real time. Basically you can request a row on an index value. You can forget joins or do them byyourselves. That’s mean that you data must be exactly formatted for your queries, and that new query types mean to change the representation of your data. More than a change, you copy it because you want to be able to perform both queries. Having multiple copy of same information with eventual consistency isn’t great. Map/Reduce? This is not something for online transactions. Map/Reduce is equivalent of full table scan and can need several hours/weeks before retriving the result. Map/Reduce is usefull for statistics, not for realtime queries.

    MySQL vs Oracle DB? First they are both owned by Oracle and you must pay for both for commercial use.

    Second MySQL is very basic. Basic queries, basic features. I agree this just fit many application need. And then this is perfect. Go for it… or go for Postgres as this one is a REAL open source DB, not one you have to pay for commercial use.

    But if you need advenced features, Oracle is a very competitive offer. You gain on developpement time, it allow some architecture that are simply not possible with MySQL.

    You can view Oracle like an plane. It cost a lost of fuel, but it can land you to other continents fast. MySQL is more like a car. It is less expensive, there are less things to learn to drive it but it is more suited to short trips than long ones.

  15. BTW, Why the title is misleading !!,
    It gives impression that your company has complicated then the Twitter/FB

    I believe it to be a trick to get attention !

  16. dear @sn

    please read the title and the post carefully
    title says “Our Companies databases are more complicated than twitter and facebook”

    and the article talks about the databases and DBMSes used in the companies in our region !

    hope this clearify your mistake

  17. So that the CIO of the company can swing his Misbah around his finger when he replies about what database he is using.

  18. Have you looked at those different features of the database? Have you look at how Facebook do away with MySQL? They consider every single column, every single type, what is the frequency of usage, usage patterns and so on in order to make one decision about one piece of logic. In other words, its NOT simple. But that is okay, since they save money on that even though they paid big on engineering resources, which easily pays when you have the economy of scale.

    And yes, as other poster has point out, the data structure is very simple and they don’t use joins or transactions. Yes it is MySQL, but it is not being used as RDBMS. Since the DB does not handle data integrity, then it is left to the programmer to do it, for every cases. They log to the death, they keep more than 1TB of logs per day and they have facility to mine to log for potential problems. Would your company do that?

    Any RDBMS can be very fast when they have been optimized as they are in case of Facebook, hence that is why they don’t really need something like Oracle, since they are not using the functionality of RDBMS anyway.

    In an enterprise, the data are being accessed by multitude of applications, many of them is outside the control of who created and maintained the database. In Facebook cases they shard the database, but it is not an option for many enterprises since they cannot define how the data is accessed, which is how Facebook/Twitter/etc actually optimize it. Oracle can easily provide partitions (with some fee of course, with its pro/cons) and it is much easier to handle in the application side. This also means they don’t have to hire PhD grads to take care of their simple, boring enterprise problems, which those people would not have interest either.

    And if your companies do so choose to use NoSQL, or MySQL the NoSQL way, the moment there is a new way to access to the same data, then you need to duplicate all the data that you need to access in the new way. Again, you are responsible to make it consistent as well, since the data has not been normalized. Your data storage cost is going to increase as well, like double or triple when there is new access requirement. All this would not be necessary since most enterprise environment does not have the scale that Facebook or Twitter have, but instead has more logic, which Facebook or Twitter don’t have.

    All in all, its about trade off. You can easily make a Twitter clone in a weekend, but that doesn’t mean that 100 million people can access it without a glitch. You can also take Twitter’s approach of engineering to your enterprise application and feel good about your engineering prowess, but in the end any new features are going to slow to halt and the cost will go through the roof.

  19. @Nicolas
    I see where you are going, and i agree with you for the most part.
    I was referring to your first comment, it appeared to me that you implied that MySql has consistency and reliability issues, which is obviously not the case.

  20. Quite Simple…
    MySQL is not accountable and hence cannot be sued for any problem but when Oracle, Ms SQL on the other hand could be held responsible for any problem…

Leave a Reply