IOTA-Lambda 架构(IOTA)

作者:cmwxj@126.com
发布时间: 2019-12-28 08:33:32 (5年前)
关键词: IOTA Lambda
摘要:

                              
                                     Lambda 架构由Storm的作者Nathan Marz提出，其设计目的在于提供一个能满足大数据系统关键特性的架构，包括高容错、低延迟、可扩展等。其整合离线计算与实时计算，融合不可变性、读写分离和复杂性隔离等原则，可集成Hadoop, Kafka, Spark，Storm等各类大数据组件。

<div style="width:595.0pt;margin-bottom:72.0pt;margin-top:72.0pt;margin-left:90.0pt;margin-right:90.0pt;">
 <p style="text-align:left;"><span style="font-family:'Verdana';font-size:21.0pt;font-weight:bold;color:#7b7d62;background-color:#ffffff;">Lambda </span><span style="font-family:'Verdana';font-size:21.0pt;font-weight:bold;color:#7b7d62;background-color:#ffffff;">架构</span></p>
 <p style="text-align:left;"><span id="_GoBack"></span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Lambda </span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">架构由</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Storm</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">的作者</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Nathan </span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Marz</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">提出，其设计目的在于提供一个能满足大数据系统关键特性的架构，包括高容错、低延迟、可扩展等。其整合离线计算与实时计算，融合</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">不</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">可变性、读写分离和复杂性隔离等原则，可集成</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Hadoop, Kafka, Spark</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">，</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Storm</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">等各类大数据组件。</span></p>
 <p style="text-align:left;"><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Lambda </span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">架构可分解为三层</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">，即</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Batch Layer, Real-Time(Speed) Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">和</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Serving Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">。</span></p>
 <p style="text-align:left;"><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Batch Layer : </span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">存储数据集，在数据集上预先计算查询函数，并构建查询所对应的</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">View</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">。</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Batch Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">可以很好的处理离线数据，但有很多场景数据是不断实时生成且需要实时查询处理，对于这情况，</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;"> Speed Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">更为适合。</span></p>
 <p style="text-align:left;"><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Speed Layer : Batch Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">处理的是全体数据集，而</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Speed Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">处理的是最近的增量数据流。</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Speed Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">为了效率，在接收到新的数据后会不断更新</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Real-time View</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">，而</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Batch Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">是根据全体离线数据</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">集直接</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">得到</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Batch View</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">。</span></p>
 <p style="text-align:left;"><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Serving Layer : Serving Layer</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">用于合并</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Batch View</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">和</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">Real-time View</span><span style="font-family:'Verdana';font-size:10.0pt;color:#7b7d62;background-color:#ffffff;">中的结果数据集到最终数据集。</span></p>
 <p style="text-align:left;"><img src="/user/files/p62zu0nscKf7uxNhKcAe5Sk60PFAU74VMF3hpmeI3a0.png" width="474.75pt" height="100%"></p>
 <p style="text-align:left;"><span style="font-family:'Verdana';font-size:21.0pt;font-weight:bold;color:#7b7d62;background-color:#ffffff;">推荐系统之</span><span style="font-family:'Verdana';font-size:21.0pt;font-weight:bold;color:#7b7d62;background-color:#ffffff;">lambda</span><span style="font-family:'Verdana';font-size:21.0pt;font-weight:bold;color:#7b7d62;background-color:#ffffff;">架构</span></p>
 <p style="text-align:left;"><img src="/user/files/1lK4ADIUy9Iz4mhk4Lxxhe0uOIacI-qkBmt-Pmpmbb0.png" width="613.5pt" height="100%"></p>
 <p style="text-align:left;"><br></p>
 <p style="text-align:left;"><img src="/user/files/MrRhsv-d2gFQ6yt-qUI5mCbQ9iS2MlfU6oBKQH3Zkzg.png" width="622.5pt" height="100%"></p>
 <p style="text-align:left;"><br></p>
 <p></p>
</div>

0 条回复

动动手指，沙发就是你的了！

登录后才能参与评论