BigQuery
BigQuery is a web service from Google that belongs to the line of Analytics products. The service was specially developed for particularly fast and web-based data retrieval and for working with very large amounts of data.
General information
BigQuery works with data volumes in the range of several billion lines. A special syntax similar to SQL is used to disseminate these. To enable the BigQuery service to be used, Google uses existing cloud storage infrastructures that are decentralized, enabling fast data processing. The result is a distribution of data within a few seconds. BigQuery is implemented with a REST-oriented programming interface (API).
BigQuery was released by Google in 2011 under the name "V2". Since then, the system has gone through numerous development stages.
BigQuery for companies with large amounts of data
BigQuery is primarily intended for developers and companies who want to move and analyze large amounts of data. It was necessary to develop appropriate tools to be able to conduct analyses of Google’s indexing. While at the beginning Google only used these internally, BigQuery can be seen as a more advanced version for companies.
The technical basis of BigQuery is an Online Analytical Processing System: in short: OLAP. This allows the processing of data records in the range of billions of characters, or the size of terabytes, and is used in conjunction with Big Data. It is conceivable to use BigQuery via a command line, a REST API, or a web interface. All data can be uploaded to Google servers in CVS format.
To be able to use BigQuery, there is a price graduation, which is calculated according to storage space. The maximum amount of data is 2TB, the limit of the daily possible requests is 1000. 100GB are free of charge, further costs are calculated according to the amount of data.
BigQuery is not suitable for processing Online Transaction Processing (OLTP for short), nor for changes to stored data that are not possible.
Examples for BigQuery
Limits for large amounts of data appear can appear when it comes to search engine optimization (SEO). For example, if you want to know how the traffic of all link partnerships has developed over the last six months, you are quickly faced with sampling problems. The evaluation of keyword clusters of the last one or two years is also difficult, which can hardly be shown using the Google Analytics Interface. With BigQuery, SEO can be significantly improved.
The situation is similar if different data sources are to be analyzed, for example for operators of shops that operate both online and offline. Here the different sales data can be analyzed and compared by BigQuery.
QlikView and Google BigQuery
The BigQuery web service can be used by developers and companies without having to make additional investments in hardware or software in advance. In addition, BigQuery is easy to use, scalable, and its on-demand deployment allows powerful data analysis.
The QlikView BusinessDiscovery platform is a customizable extension that allows easy integration with Google BigQuery. With the QlikView connector, users can load BigQuery data into memory, and search for information without any restrictions. QlikView's associative data analysis also gives users the flexibility to search and edit BigQuery data.
QlikView provides an object extension in addition to the customizable connector, allowing the QlikView dashboard to connect directly to Google's BigQuery. For example, employees from specialist departments can ask ad-hoc questions and receive answers in a short time - even with large amounts of data. Even data that is not in the memory can be evaluated spontaneously. All results are already available after a few seconds, the writing of SQL lines is omitted.
The integration of QlikView and BigQuery makes it much easier to use because even users with little or no technical knowledge can evaluate billions of data lines and search for information relevant to them. The work is also possible without having to bring extensive SQL know-how.
Significance for development
BigQuery can help companies to better handle and extract large amounts of data that would not be accessible without the help of BigQuery, or only with difficulty and with considerable effort. In principle, however, the use of big data management systems requires the openness of IT departments and certain knowledge, for example with regard to systems such as Hadoop or NoSQL.
Nightly evaluation of large amounts of data becomes easier with BigQuery, and even in daily business, further measures can be taken as the information is gained quickly.
However, BigQuery is not suitable for every company. The system only makes sense if very large amounts of data accumulate, or if search engine optimization or shop operations are to be optimized and evaluated. The decisive factor is not so much the industry as the amount of data that arises. The use of big data management systems such as BigQuery, NoSQL or Hadoop is often oversized and therefore not advisable. If these requirements apply, it is sufficient to use conventional database systems. This can save you the technical and financial effort under certain circumstances.