Design and Implementation of International Meteorological Communication Data Collection and Management System
- Article
- Figures
- Metrics
- Preview PDF
- Reference
- Related
- Cited by
- Materials
Abstract:
In recent years, meteorological and climate services have increased their requirements for foreign data types and timeliness. An international meteorological communication data collection and management system based on process control is designed and implemented to solve the collection and distribution pressure and management problems of a large amount of data. The system applies the technology of Logstash, Inotify and Kibana, supports a variety of communication protocols, expands the network crawler function of Scrapy, and achieves the fast and efficient collection and integration management of a large amount of data through process control and task scheduling, effectively improving the data parallel processing ability and scalability. And the whole process of monitoring is designed to ensure data timeliness. The system has been online to provide services, providing operational/scientific research users with more than 300 kinds of global observation data, foreign satellite data and numerical prediction products; the daily data collection is about 2 TB.
Keywords:
Project Supported:
Clc Number:
You are thevisitors
Supported by:Beijing E-Tiller Technology Development Co., Ltd.
Supported by:Beijing E-Tiller Technology Development Co., Ltd.