基于流程控制的國(guó)際氣象通信數(shù)據(jù)收集管理系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)
作者:
作者單位:

作者簡(jiǎn)介:

通訊作者:

中圖分類號(hào):

基金項(xiàng)目:


Design and Implementation of International Meteorological Communication Data Collection and Management System
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 圖/表
  • |
  • 訪問(wèn)統(tǒng)計(jì)
  • |
  • 參考文獻(xiàn)
  • |
  • 相似文獻(xiàn)
  • |
  • 引證文獻(xiàn)
  • |
  • 資源附件
  • |
  • 文章評(píng)論
    摘要:

    近年來(lái),天氣和氣候業(yè)務(wù)對(duì)國(guó)外數(shù)據(jù)種類和時(shí)效要求不斷提高,為解決大量數(shù)據(jù)的收集分發(fā)壓力以及管理問(wèn)題,設(shè)計(jì)并實(shí)現(xiàn)了基于流程控制的國(guó)際氣象通信數(shù)據(jù)收集管理系統(tǒng)。系統(tǒng)應(yīng)用數(shù)據(jù)收集器Logstash、Inotify監(jiān)聽(tīng)機(jī)制與分析平臺(tái)Kibana等技術(shù),支持多種通信協(xié)議,擴(kuò)展Scrapy網(wǎng)絡(luò)爬蟲(chóng)功能,通過(guò)流程控制、任務(wù)調(diào)度等方式實(shí)現(xiàn)大量數(shù)據(jù)快速高效收集和集成管理,有效提高數(shù)據(jù)并行處理能力和可擴(kuò)展性,且設(shè)計(jì)了全流程監(jiān)視,保障數(shù)據(jù)時(shí)效。系統(tǒng)已經(jīng)上線提供服務(wù),為業(yè)務(wù)/科研用戶提供全球觀測(cè)數(shù)據(jù)、國(guó)外衛(wèi)星數(shù)據(jù)和數(shù)值預(yù)報(bào)產(chǎn)品等300余種,日收集數(shù)據(jù)量約2 TB。

    Abstract:

    In recent years, meteorological and climate services have increased their requirements for foreign data types and timeliness. An international meteorological communication data collection and management system based on process control is designed and implemented to solve the collection and distribution pressure and management problems of a large amount of data. The system applies the technology of Logstash, Inotify and Kibana, supports a variety of communication protocols, expands the network crawler function of Scrapy, and achieves the fast and efficient collection and integration management of a large amount of data through process control and task scheduling, effectively improving the data parallel processing ability and scalability. And the whole process of monitoring is designed to ensure data timeliness. The system has been online to provide services, providing operational/scientific research users with more than 300 kinds of global observation data, foreign satellite data and numerical prediction products; the daily data collection is about 2 TB.

    參考文獻(xiàn)
    相似文獻(xiàn)
    引證文獻(xiàn)
引用本文

顧文靜,王鵬,戴晴,韓鑫強(qiáng).基于流程控制的國(guó)際氣象通信數(shù)據(jù)收集管理系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)[J].氣象科技,2023,51(1):31~39

復(fù)制
分享
文章指標(biāo)
  • 點(diǎn)擊次數(shù):
  • 下載次數(shù):
  • HTML閱讀次數(shù):
  • 引用次數(shù):
歷史
  • 收稿日期:2021-11-08
  • 定稿日期:2022-10-26
  • 錄用日期:
  • 在線發(fā)布日期: 2023-03-03
  • 出版日期:
您是第位訪問(wèn)者
技術(shù)支持:北京勤云科技發(fā)展有限公司
宁陵县| 防城港市| 山丹县| 峨边| 五华县| 临沂市| 昌江| 葵青区| 台中县| 沧源| 南雄市| 连州市| 丁青县| 宣城市| 景洪市| 罗甸县| 新泰市| 陆川县| 班玛县| 故城县| 十堰市| 舞阳县| 黎城县| 左贡县| 吕梁市| 昌宁县| 福州市| 习水县| 汝州市| 衡水市| 长寿区| 绍兴县| 丁青县| 新闻| 即墨市| 遵义县| 全州县| 从江县| 游戏| 灵武市| 玉树县|