What is a time series database

Time series database - Give wings to the Internet of Things - HBase/Spark/BigData with attitude

Overall introduction

Apache IoTDB started at the School of Software at Tsinghua University and is a time series database. The main usage scenario isInternet of ThingsRelated industries, such as: Internet of Vehicles, Wind Power, Subway, Aircraft Monitoring, etc., you can view specific application cases and company details:IoTDB usage information collection in actual companies. It adopts columnar storage, data encoding, pre-computing and indexing technology, has a SQL-like interface that can support writing millions of data points per node per second, and can obtain query results of more than trillions of data points in seconds. It can also be easily integrated with Apache Hadoop, MapReduce, and Apache Spark for analysis.

Features of IoT data collection

The Internet of Things is characterized by the existence of one or more devices, which are organized together in various forms to observe or record data generated by the same environment at the same time.

The Past and Present Life of Apache IoTDB

In 2012, Sany Heavy Industry's actual business had 200,000 equipment stored data for 3 years, and TB-level data made Oracle unable to bear it. The key problem is not just the large stock data, but the new data is still growing at a very fast rate. Later the company contacted IoTDB The first batch of developers, but the plan at that time was still based on Cassandra. At that time, a cluster of 5 machines was planned, and the performance was just met, but as time went by, the total number of equipment was increasing, and the number of query requests for the business system was increasing.

In 2015, we began to develop a distributed timing database based on Cassandra, but because it was not completely self-developed, it was a bit restricted. Moreover, after a lot of effort, Cassandra finally found that if it was modified, it might require large-scale reconstruction of the code of Cassandra data. Finally, we decided to redesign a storage method to solve the efficient writing of timing data, low-latency reading, and high compression ratio persistence in the Internet of Things scenario. The IoTDB project began to be born and embarked on the road of independent research and development. Later, the IoTDB project was donated to the Apache Foundation project for incubation. Later, after graduation, it developed into the current Apache IoTDB.

System architecture

The IoTDB suite consists of several components, jointly forming a series of functions such as "data collection-data writing-data storage-data query-data visualization-data analysis". The following figure shows the overall application architecture formed after using all components of the IoTDB suite. It is hereby said that all components form the IoTDB suite, and IoTDB specifically refers to the time series database components in it.

In the above figure, users can import time series data such as system status data collected from sensors on the device, server load and CPU memory, time series data in message queues, application time series data or other databases into local or remote IoTDB through JDBC. Users can also write the above data directly to the TsFile file locally (or on HDFS).

TsFile files can be written to HDFS, thereby realizing data processing tasks such as exception detection and machine learning on Hadoop or Spark's data processing platforms.

For TsFile files written to HDFS or local, you can use the TsFile-Hadoop or TsFile-Spark connector to allow Hadoop or Spark for data processing.

For the analysis results, they can be written back into a TsFile file.

IoTDB and TsFile also provide corresponding client tools to meet the needs of users to view and write data, such as SQL forms, script forms and graphical forms.

Performance comparison

The test tool was developed by the Big Data Laboratory of Tsinghua University

1. Comparison of write performance

The above set of data shows that the write performance is more than 10 times higher than the same database, and the writing speed of a single machine reaches 20 million per second. Moreover, the hard disk occupies the smallest, which may be 1 to 2 hard disks per month in online businesses with relatively large data.

2. Query performance comparison

Raw data query

Aggregated data query

3. Comparison chart

Overall IoTDB Whether in writing, raw data query or aggregation query, it has almost 10 times the performance of competitor databases, and the hard disk occupies 10 times less than the same database.