Antwort Is Kafka change data capture? Weitere Antworten – Is Kafka a CDC tool

Confluent Kafka is a distributed event streaming platform that enables real-time data processing and communication across various systems. It provides a reliable and scalable infrastructure for streaming, storing, and processing data and is often integrated into CDC workflows.You can use AWS DMS to migrate data to an Apache Kafka cluster. Apache Kafka is a distributed streaming platform. You can use Apache Kafka for ingesting and processing streaming data in real-time.Although it's possible to use Kafka as a data store or to create your own version of a database, Kafka isn't a database in and of itself.

How is Kafka different from a normal database : Kafka has similar properties to a database, however, Kafka stores data as a series of events while other database's co-locate data based on the keys and indexes to facilitate fast lookup. In a database key, indexes are assigned at the table or document level, whereas in Kafka, keys are assigned at the message level.

Is Kafka a ETL tool

.wDYxhc{clear:both}.cUnQKe .wDYxhc,.related-question-pair .wDYxhc,.M8OgIe .fm06If .wDYxhc{clear:none}html .dG2XIf .xpdclose .LKPcQc,html .dG2XIf .xpdopen .viOShc{padding-top:0;}.garHBe{color:var(–COEmY);font-size:20px;line-height:1.3;position:relative}.garHBe a{color:var(–COEmY)}.xpdopen .Hhmu2e{overflow:inherit}.xpdclose .a84NUc:not(:empty),.xpdopen .CQKTwc:not(:empty){border-top:solid 1px #ebebeb;border-radius:0}.xpdclose .pmYXee .a84NUc:not(:empty),.xpdopen .pmYXee .CQKTwc:not(:empty){border-top:none}.cAn2Qb .a84NUc:not(:empty),.cAn2Qb .CQKTwc:not(:empty){border-top:none}.SHa8ve{display:none}.xpdclose .TOfaBb.SHa8ve,.xpdopen .LL5J7d.SHa8ve{display:block}.rKR6H{padding-top:0.1px}.xpdclose .oHglmf,.xpdopen .xzPb7d{padding-bottom:16px}.xpdclose .kp-header .oHglmf,.xpdopen .kp-header .xzPb7d{padding-bottom:0}.c2xzTb .xpdclose .oHglmf,.c2xzTb .xpdopen .xzPb7d{padding-bottom:0}.HSryR .xpdclose .oHglmf,.HSryR .xpdopen .xzPb7d{padding-bottom:0}.Wnoohf .xpdopen .yp1CPe,.OJXvsb .xpdclose .siXlze{padding-bottom:16px}.Wnoohf .xpdopen .yp1CPe .xpdbox g-more-link{top:28px}.Wnoohf .xpdclose .wDYxhc.xsZWvb,.Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}.OJXvsb .xpdclose .wDYxhc.siXlze{padding-bottom:16px}#rhs .dG2XIf .xpdclose .LKPcQc,#rhs .dG2XIf .xpdopen .viOShc{padding-top:0}#rhs .Wnoohf .xpdopen .yp1CPe,#rhs .OJXvsb .xpdclose .siXlze{padding-bottom:15px}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb.ecRggb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh.ecRggb,#rhs .dG2XIf .ecRggb{padding-bottom:15px}.ecRggb{background-color:#fff;box-shadow:none;border:1px solid #dadce0;border-radius:8px;margin:0 0 16px 0}.Jb0Zif .UiGGAb.ILfuVd,.ILfuVd{font-size:16px;line-height:24px;}.fm06If .ILfuVd.NA6bn,.c2xzTb .ILfuVd.duSGDe,.c2xzTb .ILfuVd.NA6bn.c3biWd{font-size:14px;line-height:1.58;}.c2xzTb .zloOqf{font-size:16px;margin-bottom:16px;margin-top:-8px;}.FLWspc{margin:16px}.LFYdN{font-size:16px;margin:16px}.trNcde .ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:14px;line-height:22px}.HSryR .ILfuVd{line-height:1.5}.ss6qqb .ILfuVd,.ss6qqb .UiGGAb.ILfuVd{font-size:14px;line-height:22px}.c2xzTb .qLYAZd{margin:16px 16px 16px 0;}.d9FyLd{padding:0 0 10px}.hgKElc{padding:0 8px 0 0}.kX21rb{font-size:12px;line-height:1.34;display:inline-block;font-family:Google Sans,arial,sans-serif;padding-right:0;white-space:nowrap}.SPV4pd{display:flex;overflow-x:auto}.SPV4pd::-webkit-scrollbar{display:none}.Vjskue{margin-left:8px}.JT9o0,.ZnIJV{align-items:center;border:1px solid var(–mXZkqc);border-radius:16px;box-sizing:border-box;color:#1a73e8;cursor:pointer;display:flex;height:32px;margin-bottom:16px}.BkHX0d{font-family:Google Sans,arial,sans-serif-medium,sans-serif;font-size:14px;line-height:20px;padding-right:20px;text-align:center;white-space:nowrap}.xv1D5b{display:block;padding:0 8px 0 12px;width:18px;height:18px}.LGOjhe{overflow:hidden;padding-bottom:20px}.trNcde .LGOjhe{overflow:visible}.SALvLe .LGOjhe,.Jb0Zif .LGOjhe,.HSryR .LGOjhe,.IVvPP .LGOjhe{padding:0 0 10px;margin:10px 0 0 0}.s6JM6d .SALvLe .LGOjhe{margin-top:12px;}.s6JM6d .HSryR .LGOjhe{margin-top:12px;padding-bottom:20px}#res .Jb0Zif .qDOt0b h3,.IVvPP .qDOt0b h3{font-size:16px;line-height:1.375}.Jb0Zif .qDOt0b .iUh30,.IVvPP .qDOt0b .iUh30{font-size:14px;line-height:1.58}.IVvPP .qDOt0b{border-bottom:1px solid var(–gS5jXb);padding-bottom:6px}.rjOVwe.ILfuVd{font-size:16px;line-height:20px}.dG2XIf .rjOVwe{padding-top:22px;padding-bottom:0;margin-bottom:1px}.xWd7I{padding-top:12px;}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{font-size:14px;line-height:22px}.n6SJS .ILfuVd{font-size:14px}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{line-height:20px}.xpd-wa .g-blk{margin:initial}.xpd-wa .dG2XIf{border:initial;box-shadow:none}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd,.w5mTAe .ILfuVd,.W0urI .ILfuVd{font-family:Google Sans,arial,sans-serif;color:var(–YLNNHc)}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd{font-size:20px;line-height:28px}.cUnQKe .ILfuVd,.cUnQKe .UiGGAb.ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:16px;line-height:24px}.fm06If .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b,.c2xzTb .ILfuVd b,.w5mTAe .ILfuVd b,.W0urI .ILfuVd b{background-color:#d3e3fd;color:#040c28;font-weight:500}@media (forced-colors:active){.fm06If .ILfuVd b,.c2xzTb .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b{background-color:Mark;color:MarkText}}.IiOSLb .rsGxI.Ww4FFb,.Ww4FFb{background-color:var(–xhUGwc);border-radius:0px;border-width:0px;box-shadow:none}.Ww4FFb .mnr-c:not(:empty),.mnr-c:not(:empty) .Ww4FFb,.Ww4FFb .Ww4FFb{box-shadow:none;margin-bottom:0px}.vt6azd{margin:0px 0px 8px}.vt6azd{margin:0px 0px 30px}.CBvvz{margin:-var(–sds-sys-measurement–space-sm) 0px 0px}.CBvvz{margin:-30px 0px 0px}
Apache Kafka: A Complete Streaming ETL Platform

Kafka Streams API gives applications the stream processing capabilities to transform data, one message or event at a time. These transformations can include joining multiple data sources, filtering data, and aggregating data over a period of time.

What is CDC in Kafka : Change Data Capture (CDC) involves observing the changes happening in a database and making them available in a form that can be exploited by other systems. One of the most interesting use-cases is to make them available as a stream of events.

.wDYxhc{clear:both}.cUnQKe .wDYxhc,.related-question-pair .wDYxhc,.M8OgIe .fm06If .wDYxhc{clear:none}html .dG2XIf .xpdclose .LKPcQc,html .dG2XIf .xpdopen .viOShc{padding-top:0;}.garHBe{color:var(–COEmY);font-size:20px;line-height:1.3;position:relative}.garHBe a{color:var(–COEmY)}.xpdopen .Hhmu2e{overflow:inherit}.xpdclose .a84NUc:not(:empty),.xpdopen .CQKTwc:not(:empty){border-top:solid 1px #ebebeb;border-radius:0}.xpdclose .pmYXee .a84NUc:not(:empty),.xpdopen .pmYXee .CQKTwc:not(:empty){border-top:none}.cAn2Qb .a84NUc:not(:empty),.cAn2Qb .CQKTwc:not(:empty){border-top:none}.SHa8ve{display:none}.xpdclose .TOfaBb.SHa8ve,.xpdopen .LL5J7d.SHa8ve{display:block}.rKR6H{padding-top:0.1px}.xpdclose .oHglmf,.xpdopen .xzPb7d{padding-bottom:16px}.xpdclose .kp-header .oHglmf,.xpdopen .kp-header .xzPb7d{padding-bottom:0}.c2xzTb .xpdclose .oHglmf,.c2xzTb .xpdopen .xzPb7d{padding-bottom:0}.HSryR .xpdclose .oHglmf,.HSryR .xpdopen .xzPb7d{padding-bottom:0}.Wnoohf .xpdopen .yp1CPe,.OJXvsb .xpdclose .siXlze{padding-bottom:16px}.Wnoohf .xpdopen .yp1CPe .xpdbox g-more-link{top:28px}.Wnoohf .xpdclose .wDYxhc.xsZWvb,.Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}.OJXvsb .xpdclose .wDYxhc.siXlze{padding-bottom:16px}#rhs .dG2XIf .xpdclose .LKPcQc,#rhs .dG2XIf .xpdopen .viOShc{padding-top:0}#rhs .Wnoohf .xpdopen .yp1CPe,#rhs .OJXvsb .xpdclose .siXlze{padding-bottom:15px}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb.ecRggb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh.ecRggb,#rhs .dG2XIf .ecRggb{padding-bottom:15px}.ecRggb{background-color:#fff;box-shadow:none;border:1px solid #dadce0;border-radius:8px;margin:0 0 16px 0}.Jb0Zif .UiGGAb.ILfuVd,.ILfuVd{font-size:16px;line-height:24px;}.fm06If .ILfuVd.NA6bn,.c2xzTb .ILfuVd.duSGDe,.c2xzTb .ILfuVd.NA6bn.c3biWd{font-size:14px;line-height:1.58;}.c2xzTb .zloOqf{font-size:16px;margin-bottom:16px;margin-top:-8px;}.FLWspc{margin:16px}.LFYdN{font-size:16px;margin:16px}.trNcde .ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:14px;line-height:22px}.HSryR .ILfuVd{line-height:1.5}.ss6qqb .ILfuVd,.ss6qqb .UiGGAb.ILfuVd{font-size:14px;line-height:22px}.c2xzTb .qLYAZd{margin:16px 16px 16px 0;}.d9FyLd{padding:0 0 10px}.hgKElc{padding:0 8px 0 0}.kX21rb{font-size:12px;line-height:1.34;display:inline-block;font-family:Google Sans,arial,sans-serif;padding-right:0;white-space:nowrap}.SPV4pd{display:flex;overflow-x:auto}.SPV4pd::-webkit-scrollbar{display:none}.Vjskue{margin-left:8px}.JT9o0,.ZnIJV{align-items:center;border:1px solid var(–mXZkqc);border-radius:16px;box-sizing:border-box;color:#1a73e8;cursor:pointer;display:flex;height:32px;margin-bottom:16px}.BkHX0d{font-family:Google Sans,arial,sans-serif-medium,sans-serif;font-size:14px;line-height:20px;padding-right:20px;text-align:center;white-space:nowrap}.xv1D5b{display:block;padding:0 8px 0 12px;width:18px;height:18px}.LGOjhe{overflow:hidden;padding-bottom:20px}.trNcde .LGOjhe{overflow:visible}.SALvLe .LGOjhe,.Jb0Zif .LGOjhe,.HSryR .LGOjhe,.IVvPP .LGOjhe{padding:0 0 10px;margin:10px 0 0 0}.s6JM6d .SALvLe .LGOjhe{margin-top:12px;}.s6JM6d .HSryR .LGOjhe{margin-top:12px;padding-bottom:20px}#res .Jb0Zif .qDOt0b h3,.IVvPP .qDOt0b h3{font-size:16px;line-height:1.375}.Jb0Zif .qDOt0b .iUh30,.IVvPP .qDOt0b .iUh30{font-size:14px;line-height:1.58}.IVvPP .qDOt0b{border-bottom:1px solid var(–gS5jXb);padding-bottom:6px}.rjOVwe.ILfuVd{font-size:16px;line-height:20px}.dG2XIf .rjOVwe{padding-top:22px;padding-bottom:0;margin-bottom:1px}.xWd7I{padding-top:12px;}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{font-size:14px;line-height:22px}.n6SJS .ILfuVd{font-size:14px}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{line-height:20px}.xpd-wa .g-blk{margin:initial}.xpd-wa .dG2XIf{border:initial;box-shadow:none}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd,.w5mTAe .ILfuVd,.W0urI .ILfuVd{font-family:Google Sans,arial,sans-serif;color:var(–YLNNHc)}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd{font-size:20px;line-height:28px}.cUnQKe .ILfuVd,.cUnQKe .UiGGAb.ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:16px;line-height:24px}.fm06If .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b,.c2xzTb .ILfuVd b,.w5mTAe .ILfuVd b,.W0urI .ILfuVd b{background-color:#d3e3fd;color:#040c28;font-weight:500}@media (forced-colors:active){.fm06If .ILfuVd b,.c2xzTb .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b{background-color:Mark;color:MarkText}}.IiOSLb .rsGxI.Ww4FFb,.Ww4FFb{background-color:var(–xhUGwc);border-radius:0px;border-width:0px;box-shadow:none}.Ww4FFb .mnr-c:not(:empty),.mnr-c:not(:empty) .Ww4FFb,.Ww4FFb .Ww4FFb{box-shadow:none;margin-bottom:0px}.vt6azd{margin:0px 0px 8px}.vt6azd{margin:0px 0px 30px}.CBvvz{margin:-var(–sds-sys-measurement–space-sm) 0px 0px}.CBvvz{margin:-30px 0px 0px}
Apache Kafka is a distributed streaming platform to build near real-time data pipelines and streaming ETL applications. With its high throughput, low latency, and scalable architecture, Kafka is well-suited for processing and transforming large volumes of data in near real time.

.c2xzTb .kno-mrg,.ruTcId .kno-mrg{margin-bottom:24px;padding:0}.kno-mrg{position:relative;overflow:hidden}.SozcC{padding:0 0 16px 20px}#rhs .kno-mrg{border-top-right-radius:8px}#rhs .kno-mrg:not(.kno-mrg-si){border-top-left-radius:8px}.kno-mrg .wDYxhc{display:inline}.kno-mrg-si{float:right}.ss6qqb .kno-mrg{margin:0 16px 16px 16px}.M8OgIe .kno-mrg-si,.s6JM6d .kno-mrg-si{margin-left:20px;padding:0}.c2xzTb .kno-mrg-si,.ruTcId .kno-mrg-si{margin-bottom:16px}.c2xzTb .kno-mrg-si{border-top-right-radius:8px;overflow:hidden}#rhs .ss6qqb .kno-mrg{margin:0 0 16px 0}#rhs .u7yw9 .kno-mrg{margin:0}.EgHeA{margin:0 16px;padding-top:16px}.GHGvIc{border-radius:8px;line-height:8px;overflow:hidden;position:relative}.xte2qe{display:flex}.OXEsB{border-radius:8px;overflow:hidden}.OXEsB .Xn9Tkc{border-top-left-radius:8px}.OXEsB .oGwWse{border-top-right-radius:8px}.OXEsB .y0jvId{border-bottom-right-radius:8px}.OXEsB .lM9tvf{border-bottom-left-radius:8px}.o6uAG{border-radius:12px;overflow:hidden}.o6uAG .Xn9Tkc{border-top-left-radius:12px}.o6uAG .oGwWse{border-top-right-radius:12px}.o6uAG .y0jvId{border-bottom-right-radius:12px}.o6uAG .lM9tvf{border-bottom-left-radius:12px}.l5X1Ye{border-radius:16px;overflow:hidden}.l5X1Ye .Xn9Tkc{border-top-left-radius:16px}.l5X1Ye .oGwWse{border-top-right-radius:16px}.l5X1Ye .y0jvId{border-bottom-right-radius:16px}.l5X1Ye .lM9tvf{border-bottom-left-radius:16px}.HdBr8{overflow:visible}#rhs .LEsW6e .RYdNQb{margin-right:15px}.eA0Zlc{vertical-align:top;display:inline-block;margin:0;position:relative}html.zAoYTe .eA0Zlc:focus-within{outline:-webkit-focus-ring-color auto 1px}.JX86yc{display:inline-block}.eA0Zlc:not(.JX86yc){overflow:hidden}.c2xzTb .JX86yc:focus-within{outline:none}html.zAoYTe .c2xzTb .JX86yc a:focus{outline-offset:-1px;z-index:100;position:relative}.M8OgIe .eA0Zlc.qN5nNb{margin-right:4px;margin-bottom:4px}.M8OgIe .ifM9O .GMCzAd,.hvs42d .GMCzAd{border-radius:0}.WghbWd{cursor:pointer}.eA0Zlc.qN5nNb{margin-right:2px;margin-bottom:2px}.eA0Zlc.ITO9Cc{margin-right:2px;margin-bottom:2px}.eA0Zlc.LyIfQ{margin-right:10px;margin-bottom:10px}.oRnkrf.LyIfQ{margin-bottom:10px}.eA0Zlc.U1K9Vc{padding:10px;overflow:visible}.eA0Zlc.U1K9Vc .srrRv{background-color:transparent;border:2px solid;border-color:#1a73e8;border-radius:16px;box-sizing:border-box;pointer-events:none;height:100%;left:0;opacity:1;position:absolute;top:0;width:100%}.eA0Zlc.U1K9Vc .ywd3Gc.srrRv{width:calc(100% – 16px);height:calc(100% – 16px);border-radius:14px;left:8px;top:8px}.eA0Zlc.U1K9Vc:focus-within{outline:none}.eA0Zlc.bi-io{border-bottom:1px solid #fff;border-right:1px solid #fff;left:0;top:0;z-index:1}g-inner-card.xIfh4d{margin-right:4px;position:relative;border-radius:0;border:none;}div.eA0Zlc:first-child g-inner-card.xIfh4d,div.eA0Zlc:first-child g-inner-card.xIfh4d video{-webkit-border-top-left-radius:8px;-webkit-border-bottom-left-radius:8px;}div.eA0Zlc:last-of-type g-inner-card.xIfh4d,div.eA0Zlc:last-of-type g-inner-card.xIfh4d video{-webkit-border-top-right-radius:8px;-webkit-border-bottom-right-radius:8px;}.s6JM6d .ifM9O .GMCzAd,.cUnQKe .GMCzAd{border-radius:12px;margin:16px}.cUnQKe .GMCzAd{margin:0;}.c2xzTb .ifM9O .GMCzAd{margin:0}.XzTjhb .fm06If .ifM9O .GMCzAd{margin:0}.M8OgIe .XzTjhb .fm06If .ifM9O .GMCzAd{margin:4px 4px 0 0}.f8yzVb{width:calc(50% – 10px);border-radius:8px}.PEPRs{width:calc(25% – 20px);border-radius:16px}.EfhPkd{height:100%;left:0;position:absolute;top:0;width:100%;z-index:100}.rkPjCb{background-color:#fff;border-radius:16px;box-shadow:0 1px 6px 0 rgba(32,33,36,0.28);cursor:pointer;display:none;height:fit-content;padding:8px 8px 12px 8px;position:absolute;top:0;width:fit-content;z-index:102}.RjYw8b{background-image:linear-gradient(-180deg,rgba(0,0,0,0) 50%,rgba(0,0,0,.8) 100%);position:absolute;width:100%;height:100%;left:0;top:0}.eii6dc{background:rgba(0,0,0,.54);border-radius:16px;position:absolute;top:8px;right:8px;height:32px;width:32px}.HyRcxe{padding:15px 15px}.zgRAGb{position:absolute;top:0;right:0}.Ib6ph{position:absolute;top:0;right:1px}.TxERsb{display:flex;justify-content:center}.b1Zhoc{display:none}.BfmEZb{box-sizing:border-box;height:100%;left:0;pointer-events:none;position:absolute;top:0;width:100%}.FnEtTd .srrRv{border:2px solid;border-color:#1a73e8}.XOEbc img{transition:transform 200ms}.dB3j8:not(.qyKxnc) .XOEbc img,html.zAoYTe .FnEtTd:focus-within .XOEbc img{transform:scale(1.15)}.FnEtTd:focus-within{outline:none}html.zAoYTe .FnEtTd:focus-within .BfmEZb{border:2px solid;border-color:#1558d6}.uhHOwf{position:relative}.uhHOwf::after{background-color:rgba(0,0,0,0.03);bottom:0;content:”;display:block;left:0;pointer-events:none;position:absolute;right:0;top:0}.uhHOwf img{display:block}.BYbUcd{overflow:hidden;}.BYbUcd img{height:100%;width:100%}.B68O1d{background-color:rgba(0,0,0,0.54)}

As Kafka is designed to handle high volumes of data, it's overkill if you need to process only a small amount of messages per day (up to several thousand). Use traditional message queues such as RabbitMQ for relatively smaller data sets or as a dedicated task queue.

Does Kafka retain data

.wDYxhc{clear:both}.cUnQKe .wDYxhc,.related-question-pair .wDYxhc,.M8OgIe .fm06If .wDYxhc{clear:none}html .dG2XIf .xpdclose .LKPcQc,html .dG2XIf .xpdopen .viOShc{padding-top:0;}.garHBe{color:var(–COEmY);font-size:20px;line-height:1.3;position:relative}.garHBe a{color:var(–COEmY)}.xpdopen .Hhmu2e{overflow:inherit}.xpdclose .a84NUc:not(:empty),.xpdopen .CQKTwc:not(:empty){border-top:solid 1px #ebebeb;border-radius:0}.xpdclose .pmYXee .a84NUc:not(:empty),.xpdopen .pmYXee .CQKTwc:not(:empty){border-top:none}.cAn2Qb .a84NUc:not(:empty),.cAn2Qb .CQKTwc:not(:empty){border-top:none}.SHa8ve{display:none}.xpdclose .TOfaBb.SHa8ve,.xpdopen .LL5J7d.SHa8ve{display:block}.rKR6H{padding-top:0.1px}.xpdclose .oHglmf,.xpdopen .xzPb7d{padding-bottom:16px}.xpdclose .kp-header .oHglmf,.xpdopen .kp-header .xzPb7d{padding-bottom:0}.c2xzTb .xpdclose .oHglmf,.c2xzTb .xpdopen .xzPb7d{padding-bottom:0}.HSryR .xpdclose .oHglmf,.HSryR .xpdopen .xzPb7d{padding-bottom:0}.Wnoohf .xpdopen .yp1CPe,.OJXvsb .xpdclose .siXlze{padding-bottom:16px}.Wnoohf .xpdopen .yp1CPe .xpdbox g-more-link{top:28px}.Wnoohf .xpdclose .wDYxhc.xsZWvb,.Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}.OJXvsb .xpdclose .wDYxhc.siXlze{padding-bottom:16px}#rhs .dG2XIf .xpdclose .LKPcQc,#rhs .dG2XIf .xpdopen .viOShc{padding-top:0}#rhs .Wnoohf .xpdopen .yp1CPe,#rhs .OJXvsb .xpdclose .siXlze{padding-bottom:15px}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb.ecRggb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh.ecRggb,#rhs .dG2XIf .ecRggb{padding-bottom:15px}.ecRggb{background-color:#fff;box-shadow:none;border:1px solid #dadce0;border-radius:8px;margin:0 0 16px 0}.Jb0Zif .UiGGAb.ILfuVd,.ILfuVd{font-size:16px;line-height:24px;}.fm06If .ILfuVd.NA6bn,.c2xzTb .ILfuVd.duSGDe,.c2xzTb .ILfuVd.NA6bn.c3biWd{font-size:14px;line-height:1.58;}.c2xzTb .zloOqf{font-size:16px;margin-bottom:16px;margin-top:-8px;}.FLWspc{margin:16px}.LFYdN{font-size:16px;margin:16px}.trNcde .ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:14px;line-height:22px}.HSryR .ILfuVd{line-height:1.5}.ss6qqb .ILfuVd,.ss6qqb .UiGGAb.ILfuVd{font-size:14px;line-height:22px}.c2xzTb .qLYAZd{margin:16px 16px 16px 0;}.d9FyLd{padding:0 0 10px}.hgKElc{padding:0 8px 0 0}.kX21rb{font-size:12px;line-height:1.34;display:inline-block;font-family:Google Sans,arial,sans-serif;padding-right:0;white-space:nowrap}.SPV4pd{display:flex;overflow-x:auto}.SPV4pd::-webkit-scrollbar{display:none}.Vjskue{margin-left:8px}.JT9o0,.ZnIJV{align-items:center;border:1px solid var(–mXZkqc);border-radius:16px;box-sizing:border-box;color:#1a73e8;cursor:pointer;display:flex;height:32px;margin-bottom:16px}.BkHX0d{font-family:Google Sans,arial,sans-serif-medium,sans-serif;font-size:14px;line-height:20px;padding-right:20px;text-align:center;white-space:nowrap}.xv1D5b{display:block;padding:0 8px 0 12px;width:18px;height:18px}.LGOjhe{overflow:hidden;padding-bottom:20px}.trNcde .LGOjhe{overflow:visible}.SALvLe .LGOjhe,.Jb0Zif .LGOjhe,.HSryR .LGOjhe,.IVvPP .LGOjhe{padding:0 0 10px;margin:10px 0 0 0}.s6JM6d .SALvLe .LGOjhe{margin-top:12px;}.s6JM6d .HSryR .LGOjhe{margin-top:12px;padding-bottom:20px}#res .Jb0Zif .qDOt0b h3,.IVvPP .qDOt0b h3{font-size:16px;line-height:1.375}.Jb0Zif .qDOt0b .iUh30,.IVvPP .qDOt0b .iUh30{font-size:14px;line-height:1.58}.IVvPP .qDOt0b{border-bottom:1px solid var(–gS5jXb);padding-bottom:6px}.rjOVwe.ILfuVd{font-size:16px;line-height:20px}.dG2XIf .rjOVwe{padding-top:22px;padding-bottom:0;margin-bottom:1px}.xWd7I{padding-top:12px;}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{font-size:14px;line-height:22px}.n6SJS .ILfuVd{font-size:14px}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{line-height:20px}.xpd-wa .g-blk{margin:initial}.xpd-wa .dG2XIf{border:initial;box-shadow:none}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd,.w5mTAe .ILfuVd,.W0urI .ILfuVd{font-family:Google Sans,arial,sans-serif;color:var(–YLNNHc)}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd{font-size:20px;line-height:28px}.cUnQKe .ILfuVd,.cUnQKe .UiGGAb.ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:16px;line-height:24px}.fm06If .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b,.c2xzTb .ILfuVd b,.w5mTAe .ILfuVd b,.W0urI .ILfuVd b{background-color:#d3e3fd;color:#040c28;font-weight:500}@media (forced-colors:active){.fm06If .ILfuVd b,.c2xzTb .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b{background-color:Mark;color:MarkText}}.IiOSLb .rsGxI.Ww4FFb,.Ww4FFb{background-color:var(–xhUGwc);border-radius:0px;border-width:0px;box-shadow:none}.Ww4FFb .mnr-c:not(:empty),.mnr-c:not(:empty) .Ww4FFb,.Ww4FFb .Ww4FFb{box-shadow:none;margin-bottom:0px}.vt6azd{margin:0px 0px 8px}.vt6azd{margin:0px 0px 30px}.CBvvz{margin:-var(–sds-sys-measurement–space-sm) 0px 0px}.CBvvz{margin:-30px 0px 0px}
Instead of overwriting or modifying data, Kafka retains a complete history of events in the order they were received. Each event, or message, is appended to the end of the log, creating an immutable sequence of records. This log-based structure ensures that events are never modified, preserving data integrity..wDYxhc{clear:both}.cUnQKe .wDYxhc,.related-question-pair .wDYxhc,.M8OgIe .fm06If .wDYxhc{clear:none}html .dG2XIf .xpdclose .LKPcQc,html .dG2XIf .xpdopen .viOShc{padding-top:0;}.garHBe{color:var(–COEmY);font-size:20px;line-height:1.3;position:relative}.garHBe a{color:var(–COEmY)}.xpdopen .Hhmu2e{overflow:inherit}.xpdclose .a84NUc:not(:empty),.xpdopen .CQKTwc:not(:empty){border-top:solid 1px #ebebeb;border-radius:0}.xpdclose .pmYXee .a84NUc:not(:empty),.xpdopen .pmYXee .CQKTwc:not(:empty){border-top:none}.cAn2Qb .a84NUc:not(:empty),.cAn2Qb .CQKTwc:not(:empty){border-top:none}.SHa8ve{display:none}.xpdclose .TOfaBb.SHa8ve,.xpdopen .LL5J7d.SHa8ve{display:block}.rKR6H{padding-top:0.1px}.xpdclose .oHglmf,.xpdopen .xzPb7d{padding-bottom:16px}.xpdclose .kp-header .oHglmf,.xpdopen .kp-header .xzPb7d{padding-bottom:0}.c2xzTb .xpdclose .oHglmf,.c2xzTb .xpdopen .xzPb7d{padding-bottom:0}.HSryR .xpdclose .oHglmf,.HSryR .xpdopen .xzPb7d{padding-bottom:0}.Wnoohf .xpdopen .yp1CPe,.OJXvsb .xpdclose .siXlze{padding-bottom:16px}.Wnoohf .xpdopen .yp1CPe .xpdbox g-more-link{top:28px}.Wnoohf .xpdclose .wDYxhc.xsZWvb,.Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}.OJXvsb .xpdclose .wDYxhc.siXlze{padding-bottom:16px}#rhs .dG2XIf .xpdclose .LKPcQc,#rhs .dG2XIf .xpdopen .viOShc{padding-top:0}#rhs .Wnoohf .xpdopen .yp1CPe,#rhs .OJXvsb .xpdclose .siXlze{padding-bottom:15px}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb.ecRggb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh.ecRggb,#rhs .dG2XIf .ecRggb{padding-bottom:15px}.ecRggb{background-color:#fff;box-shadow:none;border:1px solid #dadce0;border-radius:8px;margin:0 0 16px 0}.Jb0Zif .UiGGAb.ILfuVd,.ILfuVd{font-size:16px;line-height:24px;}.fm06If .ILfuVd.NA6bn,.c2xzTb .ILfuVd.duSGDe,.c2xzTb .ILfuVd.NA6bn.c3biWd{font-size:14px;line-height:1.58;}.c2xzTb .zloOqf{font-size:16px;margin-bottom:16px;margin-top:-8px;}.FLWspc{margin:16px}.LFYdN{font-size:16px;margin:16px}.trNcde .ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:14px;line-height:22px}.HSryR .ILfuVd{line-height:1.5}.ss6qqb .ILfuVd,.ss6qqb .UiGGAb.ILfuVd{font-size:14px;line-height:22px}.c2xzTb .qLYAZd{margin:16px 16px 16px 0;}.d9FyLd{padding:0 0 10px}.hgKElc{padding:0 8px 0 0}.kX21rb{font-size:12px;line-height:1.34;display:inline-block;font-family:Google Sans,arial,sans-serif;padding-right:0;white-space:nowrap}.SPV4pd{display:flex;overflow-x:auto}.SPV4pd::-webkit-scrollbar{display:none}.Vjskue{margin-left:8px}.JT9o0,.ZnIJV{align-items:center;border:1px solid var(–mXZkqc);border-radius:16px;box-sizing:border-box;color:#1a73e8;cursor:pointer;display:flex;height:32px;margin-bottom:16px}.BkHX0d{font-family:Google Sans,arial,sans-serif-medium,sans-serif;font-size:14px;line-height:20px;padding-right:20px;text-align:center;white-space:nowrap}.xv1D5b{display:block;padding:0 8px 0 12px;width:18px;height:18px}.LGOjhe{overflow:hidden;padding-bottom:20px}.trNcde .LGOjhe{overflow:visible}.SALvLe .LGOjhe,.Jb0Zif .LGOjhe,.HSryR .LGOjhe,.IVvPP .LGOjhe{padding:0 0 10px;margin:10px 0 0 0}.s6JM6d .SALvLe .LGOjhe{margin-top:12px;}.s6JM6d .HSryR .LGOjhe{margin-top:12px;padding-bottom:20px}#res .Jb0Zif .qDOt0b h3,.IVvPP .qDOt0b h3{font-size:16px;line-height:1.375}.Jb0Zif .qDOt0b .iUh30,.IVvPP .qDOt0b .iUh30{font-size:14px;line-height:1.58}.IVvPP .qDOt0b{border-bottom:1px solid var(–gS5jXb);padding-bottom:6px}.rjOVwe.ILfuVd{font-size:16px;line-height:20px}.dG2XIf .rjOVwe{padding-top:22px;padding-bottom:0;margin-bottom:1px}.xWd7I{padding-top:12px;}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{font-size:14px;line-height:22px}.n6SJS .ILfuVd{font-size:14px}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{line-height:20px}.xpd-wa .g-blk{margin:initial}.xpd-wa .dG2XIf{border:initial;box-shadow:none}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd,.w5mTAe .ILfuVd,.W0urI .ILfuVd{font-family:Google Sans,arial,sans-serif;color:var(–YLNNHc)}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd{font-size:20px;line-height:28px}.cUnQKe .ILfuVd,.cUnQKe .UiGGAb.ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:16px;line-height:24px}.fm06If .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b,.c2xzTb .ILfuVd b,.w5mTAe .ILfuVd b,.W0urI .ILfuVd b{background-color:#d3e3fd;color:#040c28;font-weight:500}@media (forced-colors:active){.fm06If .ILfuVd b,.c2xzTb .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b{background-color:Mark;color:MarkText}}.IiOSLb .rsGxI.Ww4FFb,.Ww4FFb{background-color:var(–xhUGwc);border-radius:0px;border-width:0px;box-shadow:none}.Ww4FFb .mnr-c:not(:empty),.mnr-c:not(:empty) .Ww4FFb,.Ww4FFb .Ww4FFb{box-shadow:none;margin-bottom:0px}.vt6azd{margin:0px 0px 8px}.vt6azd{margin:0px 0px 30px}.CBvvz{margin:-var(–sds-sys-measurement–space-sm) 0px 0px}.CBvvz{margin:-30px 0px 0px}
In contrast to Batch ETL, Kafka enables ETL pipelines that are streaming, allowing real-time transformation and distribution of data at large scale.Kafka uses a pull-based system that allows users to request messages. Pushing is just extra work for the broker. With Kafka, the responsibility of fetching messages is on consumers. Consumers can decide at what rate they want to process the messages.

replication and cdc are totally different. Replication is used to populate the data from one database to other the database. But in CDC it captures both old and new data and this can be used for auditing purpose.

What Kafka Cannot be used for : Kafka is not a deterministic system. Safety-critical applications cannot use it for a car engine control system, a medical system such as a heart pacemaker, or an industrial process controller. A few examples where Kafka CANNOT be used for: Safety-critical data processing in the car or vehicle.

Why use Kafka instead of rest : .wDYxhc{clear:both}.cUnQKe .wDYxhc,.related-question-pair .wDYxhc,.M8OgIe .fm06If .wDYxhc{clear:none}html .dG2XIf .xpdclose .LKPcQc,html .dG2XIf .xpdopen .viOShc{padding-top:0;}.garHBe{color:var(–COEmY);font-size:20px;line-height:1.3;position:relative}.garHBe a{color:var(–COEmY)}.xpdopen .Hhmu2e{overflow:inherit}.xpdclose .a84NUc:not(:empty),.xpdopen .CQKTwc:not(:empty){border-top:solid 1px #ebebeb;border-radius:0}.xpdclose .pmYXee .a84NUc:not(:empty),.xpdopen .pmYXee .CQKTwc:not(:empty){border-top:none}.cAn2Qb .a84NUc:not(:empty),.cAn2Qb .CQKTwc:not(:empty){border-top:none}.SHa8ve{display:none}.xpdclose .TOfaBb.SHa8ve,.xpdopen .LL5J7d.SHa8ve{display:block}.rKR6H{padding-top:0.1px}.xpdclose .oHglmf,.xpdopen .xzPb7d{padding-bottom:16px}.xpdclose .kp-header .oHglmf,.xpdopen .kp-header .xzPb7d{padding-bottom:0}.c2xzTb .xpdclose .oHglmf,.c2xzTb .xpdopen .xzPb7d{padding-bottom:0}.HSryR .xpdclose .oHglmf,.HSryR .xpdopen .xzPb7d{padding-bottom:0}.Wnoohf .xpdopen .yp1CPe,.OJXvsb .xpdclose .siXlze{padding-bottom:16px}.Wnoohf .xpdopen .yp1CPe .xpdbox g-more-link{top:28px}.Wnoohf .xpdclose .wDYxhc.xsZWvb,.Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}.OJXvsb .xpdclose .wDYxhc.siXlze{padding-bottom:16px}#rhs .dG2XIf .xpdclose .LKPcQc,#rhs .dG2XIf .xpdopen .viOShc{padding-top:0}#rhs .Wnoohf .xpdopen .yp1CPe,#rhs .OJXvsb .xpdclose .siXlze{padding-bottom:15px}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb.ecRggb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh.ecRggb,#rhs .dG2XIf .ecRggb{padding-bottom:15px}.ecRggb{background-color:#fff;box-shadow:none;border:1px solid #dadce0;border-radius:8px;margin:0 0 16px 0}.Jb0Zif .UiGGAb.ILfuVd,.ILfuVd{font-size:16px;line-height:24px;}.fm06If .ILfuVd.NA6bn,.c2xzTb .ILfuVd.duSGDe,.c2xzTb .ILfuVd.NA6bn.c3biWd{font-size:14px;line-height:1.58;}.c2xzTb .zloOqf{font-size:16px;margin-bottom:16px;margin-top:-8px;}.FLWspc{margin:16px}.LFYdN{font-size:16px;margin:16px}.trNcde .ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:14px;line-height:22px}.HSryR .ILfuVd{line-height:1.5}.ss6qqb .ILfuVd,.ss6qqb .UiGGAb.ILfuVd{font-size:14px;line-height:22px}.c2xzTb .qLYAZd{margin:16px 16px 16px 0;}.d9FyLd{padding:0 0 10px}.hgKElc{padding:0 8px 0 0}.kX21rb{font-size:12px;line-height:1.34;display:inline-block;font-family:Google Sans,arial,sans-serif;padding-right:0;white-space:nowrap}.SPV4pd{display:flex;overflow-x:auto}.SPV4pd::-webkit-scrollbar{display:none}.Vjskue{margin-left:8px}.JT9o0,.ZnIJV{align-items:center;border:1px solid var(–mXZkqc);border-radius:16px;box-sizing:border-box;color:#1a73e8;cursor:pointer;display:flex;height:32px;margin-bottom:16px}.BkHX0d{font-family:Google Sans,arial,sans-serif-medium,sans-serif;font-size:14px;line-height:20px;padding-right:20px;text-align:center;white-space:nowrap}.xv1D5b{display:block;padding:0 8px 0 12px;width:18px;height:18px}.LGOjhe{overflow:hidden;padding-bottom:20px}.trNcde .LGOjhe{overflow:visible}.SALvLe .LGOjhe,.Jb0Zif .LGOjhe,.HSryR .LGOjhe,.IVvPP .LGOjhe{padding:0 0 10px;margin:10px 0 0 0}.s6JM6d .SALvLe .LGOjhe{margin-top:12px;}.s6JM6d .HSryR .LGOjhe{margin-top:12px;padding-bottom:20px}#res .Jb0Zif .qDOt0b h3,.IVvPP .qDOt0b h3{font-size:16px;line-height:1.375}.Jb0Zif .qDOt0b .iUh30,.IVvPP .qDOt0b .iUh30{font-size:14px;line-height:1.58}.IVvPP .qDOt0b{border-bottom:1px solid var(–gS5jXb);padding-bottom:6px}.rjOVwe.ILfuVd{font-size:16px;line-height:20px}.dG2XIf .rjOVwe{padding-top:22px;padding-bottom:0;margin-bottom:1px}.xWd7I{padding-top:12px;}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{font-size:14px;line-height:22px}.n6SJS .ILfuVd{font-size:14px}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{line-height:20px}.xpd-wa .g-blk{margin:initial}.xpd-wa .dG2XIf{border:initial;box-shadow:none}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd,.w5mTAe .ILfuVd,.W0urI .ILfuVd{font-family:Google Sans,arial,sans-serif;color:var(–YLNNHc)}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd{font-size:20px;line-height:28px}.cUnQKe .ILfuVd,.cUnQKe .UiGGAb.ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:16px;line-height:24px}.fm06If .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b,.c2xzTb .ILfuVd b,.w5mTAe .ILfuVd b,.W0urI .ILfuVd b{background-color:#d3e3fd;color:#040c28;font-weight:500}@media (forced-colors:active){.fm06If .ILfuVd b,.c2xzTb .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b{background-color:Mark;color:MarkText}}.IiOSLb .rsGxI.Ww4FFb,.Ww4FFb{background-color:var(–xhUGwc);border-radius:0px;border-width:0px;box-shadow:none}.Ww4FFb .mnr-c:not(:empty),.mnr-c:not(:empty) .Ww4FFb,.Ww4FFb .Ww4FFb{box-shadow:none;margin-bottom:0px}.vt6azd{margin:0px 0px 8px}.vt6azd{margin:0px 0px 30px}.CBvvz{margin:-var(–sds-sys-measurement–space-sm) 0px 0px}.CBvvz{margin:-30px 0px 0px}
Kafka: Kafka is optimized for low-latency event streaming and can provide real-time data processing capabilities. REST: REST can introduce higher latency, especially when dealing with complex requests or when a large number of requests are made due to its request-response nature.

Does Netflix use Kafka

Netflix embraces Apache Kafka® as the de-facto standard for its eventing, messaging, and stream processing needs. Kafka acts as a bridge for all point-to-point and Netflix Studio wide communications.

.wDYxhc{clear:both}.cUnQKe .wDYxhc,.related-question-pair .wDYxhc,.M8OgIe .fm06If .wDYxhc{clear:none}html .dG2XIf .xpdclose .LKPcQc,html .dG2XIf .xpdopen .viOShc{padding-top:0;}.garHBe{color:var(–COEmY);font-size:20px;line-height:1.3;position:relative}.garHBe a{color:var(–COEmY)}.xpdopen .Hhmu2e{overflow:inherit}.xpdclose .a84NUc:not(:empty),.xpdopen .CQKTwc:not(:empty){border-top:solid 1px #ebebeb;border-radius:0}.xpdclose .pmYXee .a84NUc:not(:empty),.xpdopen .pmYXee .CQKTwc:not(:empty){border-top:none}.cAn2Qb .a84NUc:not(:empty),.cAn2Qb .CQKTwc:not(:empty){border-top:none}.SHa8ve{display:none}.xpdclose .TOfaBb.SHa8ve,.xpdopen .LL5J7d.SHa8ve{display:block}.rKR6H{padding-top:0.1px}.xpdclose .oHglmf,.xpdopen .xzPb7d{padding-bottom:16px}.xpdclose .kp-header .oHglmf,.xpdopen .kp-header .xzPb7d{padding-bottom:0}.c2xzTb .xpdclose .oHglmf,.c2xzTb .xpdopen .xzPb7d{padding-bottom:0}.HSryR .xpdclose .oHglmf,.HSryR .xpdopen .xzPb7d{padding-bottom:0}.Wnoohf .xpdopen .yp1CPe,.OJXvsb .xpdclose .siXlze{padding-bottom:16px}.Wnoohf .xpdopen .yp1CPe .xpdbox g-more-link{top:28px}.Wnoohf .xpdclose .wDYxhc.xsZWvb,.Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}.OJXvsb .xpdclose .wDYxhc.siXlze{padding-bottom:16px}#rhs .dG2XIf .xpdclose .LKPcQc,#rhs .dG2XIf .xpdopen .viOShc{padding-top:0}#rhs .Wnoohf .xpdopen .yp1CPe,#rhs .OJXvsb .xpdclose .siXlze{padding-bottom:15px}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb.ecRggb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh.ecRggb,#rhs .dG2XIf .ecRggb{padding-bottom:15px}.ecRggb{background-color:#fff;box-shadow:none;border:1px solid #dadce0;border-radius:8px;margin:0 0 16px 0}.Jb0Zif .UiGGAb.ILfuVd,.ILfuVd{font-size:16px;line-height:24px;}.fm06If .ILfuVd.NA6bn,.c2xzTb .ILfuVd.duSGDe,.c2xzTb .ILfuVd.NA6bn.c3biWd{font-size:14px;line-height:1.58;}.c2xzTb .zloOqf{font-size:16px;margin-bottom:16px;margin-top:-8px;}.FLWspc{margin:16px}.LFYdN{font-size:16px;margin:16px}.trNcde .ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:14px;line-height:22px}.HSryR .ILfuVd{line-height:1.5}.ss6qqb .ILfuVd,.ss6qqb .UiGGAb.ILfuVd{font-size:14px;line-height:22px}.c2xzTb .qLYAZd{margin:16px 16px 16px 0;}.d9FyLd{padding:0 0 10px}.hgKElc{padding:0 8px 0 0}.kX21rb{font-size:12px;line-height:1.34;display:inline-block;font-family:Google Sans,arial,sans-serif;padding-right:0;white-space:nowrap}.SPV4pd{display:flex;overflow-x:auto}.SPV4pd::-webkit-scrollbar{display:none}.Vjskue{margin-left:8px}.JT9o0,.ZnIJV{align-items:center;border:1px solid var(–mXZkqc);border-radius:16px;box-sizing:border-box;color:#1a73e8;cursor:pointer;display:flex;height:32px;margin-bottom:16px}.BkHX0d{font-family:Google Sans,arial,sans-serif-medium,sans-serif;font-size:14px;line-height:20px;padding-right:20px;text-align:center;white-space:nowrap}.xv1D5b{display:block;padding:0 8px 0 12px;width:18px;height:18px}.LGOjhe{overflow:hidden;padding-bottom:20px}.trNcde .LGOjhe{overflow:visible}.SALvLe .LGOjhe,.Jb0Zif .LGOjhe,.HSryR .LGOjhe,.IVvPP .LGOjhe{padding:0 0 10px;margin:10px 0 0 0}.s6JM6d .SALvLe .LGOjhe{margin-top:12px;}.s6JM6d .HSryR .LGOjhe{margin-top:12px;padding-bottom:20px}#res .Jb0Zif .qDOt0b h3,.IVvPP .qDOt0b h3{font-size:16px;line-height:1.375}.Jb0Zif .qDOt0b .iUh30,.IVvPP .qDOt0b .iUh30{font-size:14px;line-height:1.58}.IVvPP .qDOt0b{border-bottom:1px solid var(–gS5jXb);padding-bottom:6px}.rjOVwe.ILfuVd{font-size:16px;line-height:20px}.dG2XIf .rjOVwe{padding-top:22px;padding-bottom:0;margin-bottom:1px}.xWd7I{padding-top:12px;}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{font-size:14px;line-height:22px}.n6SJS .ILfuVd{font-size:14px}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{line-height:20px}.xpd-wa .g-blk{margin:initial}.xpd-wa .dG2XIf{border:initial;box-shadow:none}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd,.w5mTAe .ILfuVd,.W0urI .ILfuVd{font-family:Google Sans,arial,sans-serif;color:var(–YLNNHc)}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd{font-size:20px;line-height:28px}.cUnQKe .ILfuVd,.cUnQKe .UiGGAb.ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:16px;line-height:24px}.fm06If .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b,.c2xzTb .ILfuVd b,.w5mTAe .ILfuVd b,.W0urI .ILfuVd b{background-color:#d3e3fd;color:#040c28;font-weight:500}@media (forced-colors:active){.fm06If .ILfuVd b,.c2xzTb .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b{background-color:Mark;color:MarkText}}.IiOSLb .rsGxI.Ww4FFb,.Ww4FFb{background-color:var(–xhUGwc);border-radius:0px;border-width:0px;box-shadow:none}.Ww4FFb .mnr-c:not(:empty),.mnr-c:not(:empty) .Ww4FFb,.Ww4FFb .Ww4FFb{box-shadow:none;margin-bottom:0px}.vt6azd{margin:0px 0px 8px}.vt6azd{margin:0px 0px 30px}.CBvvz{margin:-var(–sds-sys-measurement–space-sm) 0px 0px}.CBvvz{margin:-30px 0px 0px}
Kafka retains the segment for seven days after the segment is full. So the first message in the segment is actually deleted 14 days after it was added to the log. As it was added on the first day, it stays in the segment for seven days until the segment closes and then seven more days of retention..Z0LcW{color:var(–YLNNHc);font-size:32px;line-height:1.25;margin-bottom:5px;margin-top:-3px;word-wrap:break-word}.c2xzTb .Z0LcW{line-height:1;margin-bottom:12px;margin-top:0}.XzTjhb .fm06If .Z0LcW{margin-bottom:12px;margin-top:0}.wp-ms .Z0LcW{font-size:16px;font-family:Roboto-Medium,HelveticaNeue-Medium,Helvetica Neue,sans-serif-medium,Arial,sans-serif}.HSryR .Z0LcW,.Jb0Zif .Z0LcW{font-family:unset;font-size:30px}.trNcde .Z0LcW.AZCkJd{font-family:Google Sans,arial,sans-serif}.s6JM6d .HSryR .Z0LcW{line-height:1;margin-bottom:12px;margin-top:0}.Z0LcW.k37FLe{font-size:20px}.Z0LcW.CfV8xf{font-size:48px}.Z0LcW.vMhfn{font-size:36px}.Z0LcW.FRz20e{font-size:28px}.Z0LcW.OGwxI{font-size:20px}.yxAsKe{color:var(–IXoxUe);font-size:14px}.kZ91ed{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.IZ6rdc{width:100%}.d2J77b{display:flex}.HwtpBd{padding:4px 0;padding-top:4px}.HwtpBd{padding-top:16px;padding-bottom:16px}.HwtpBd{padding-top:16px}.c2xzTb .HwtpBd.yyuVCc{padding-top:32px;padding-bottom:48px;word-wrap:break-word}.IVvPP .HwtpBd{padding-bottom:0;padding-top:0}.related-question-pair .Z0LcW:not(.AZCkJd){margin-bottom:0}.related-question-pair .Z0LcW:not(.k37FLe){font-size:20px}.related-question-pair .Z0LcW.k37FLe{font-size:16px}.n6SJS .HwtpBd{padding:0}.n6SJS .t2b5Cf{font-size:36px;margin-bottom:10px}.n6SJS .FLP8od{text-decoration:none}.HwtpBd{padding-left:16px;padding-right:16px}.trNcde .HwtpBd,.J7wutc .HwtpBd{padding-left:0;padding-right:0}.trNcde .HwtpBd{padding-top:0}.trNcde .Z0LcW{margin-top:0}.c2xzTb .HwtpBd{padding:0}.XzTjhb .fm06If .HwtpBd{padding:0}.M8OgIe .XzTjhb .fm06If .HwtpBd{font-family:Google Sans,arial,sans-serif-medium,sans-serif;padding-top:16px}.HSryR .HwtpBd{padding:0}.related-question-pair .HwtpBd{padding-left:0;padding-right:0}.wp-ms .HwtpBd{padding-left:0;padding-right:0}.wp-ms .related-question-pair .HwtpBd,.wp-ms .c2xzTb .HwtpBd,.HSryR .wp-ms .HwtpBd{padding-left:16px;padding-right:16px}.Jb0Zif .wp-ms .related-question-pair .HwtpBd,.Jb0Zif .wp-ms .c2xzTb .HwtpBd{padding-left:0;padding-right:0}.AZCkJd{font-family:Google Sans,arial,sans-serif}.qs-ic .HwtpBd{padding-left:26px;padding-right:26px}a.FLP8od{color:var(–COEmY);}.FLP8od:hover{color:var(–JKqx2)}.TQc1id .HSryR .HwtpBd{padding-left:0;padding-right:0}.GqKvT .Z0LcW{color:#fff}.GqKvT .yxAsKe{color:rgba(255,255,255,.7)}disk

Kafka stores all messages on disk and has the capability to hold immense amounts of data for long periods of time. It's not an in-memory system that loses data when a process shuts down; instead, it persists data to disk, ensuring durability and fault-tolerance.

Is Kafka ETL or ELT : .wDYxhc{clear:both}.cUnQKe .wDYxhc,.related-question-pair .wDYxhc,.M8OgIe .fm06If .wDYxhc{clear:none}html .dG2XIf .xpdclose .LKPcQc,html .dG2XIf .xpdopen .viOShc{padding-top:0;}.garHBe{color:var(–COEmY);font-size:20px;line-height:1.3;position:relative}.garHBe a{color:var(–COEmY)}.xpdopen .Hhmu2e{overflow:inherit}.xpdclose .a84NUc:not(:empty),.xpdopen .CQKTwc:not(:empty){border-top:solid 1px #ebebeb;border-radius:0}.xpdclose .pmYXee .a84NUc:not(:empty),.xpdopen .pmYXee .CQKTwc:not(:empty){border-top:none}.cAn2Qb .a84NUc:not(:empty),.cAn2Qb .CQKTwc:not(:empty){border-top:none}.SHa8ve{display:none}.xpdclose .TOfaBb.SHa8ve,.xpdopen .LL5J7d.SHa8ve{display:block}.rKR6H{padding-top:0.1px}.xpdclose .oHglmf,.xpdopen .xzPb7d{padding-bottom:16px}.xpdclose .kp-header .oHglmf,.xpdopen .kp-header .xzPb7d{padding-bottom:0}.c2xzTb .xpdclose .oHglmf,.c2xzTb .xpdopen .xzPb7d{padding-bottom:0}.HSryR .xpdclose .oHglmf,.HSryR .xpdopen .xzPb7d{padding-bottom:0}.Wnoohf .xpdopen .yp1CPe,.OJXvsb .xpdclose .siXlze{padding-bottom:16px}.Wnoohf .xpdopen .yp1CPe .xpdbox g-more-link{top:28px}.Wnoohf .xpdclose .wDYxhc.xsZWvb,.Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}.OJXvsb .xpdclose .wDYxhc.siXlze{padding-bottom:16px}#rhs .dG2XIf .xpdclose .LKPcQc,#rhs .dG2XIf .xpdopen .viOShc{padding-top:0}#rhs .Wnoohf .xpdopen .yp1CPe,#rhs .OJXvsb .xpdclose .siXlze{padding-bottom:15px}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh{padding-bottom:0}#rhs .Wnoohf .xpdclose .wDYxhc.xsZWvb.ecRggb,#rhs .Wnoohf .xpdopen .wDYxhc.EfDVh.ecRggb,#rhs .dG2XIf .ecRggb{padding-bottom:15px}.ecRggb{background-color:#fff;box-shadow:none;border:1px solid #dadce0;border-radius:8px;margin:0 0 16px 0}.Jb0Zif .UiGGAb.ILfuVd,.ILfuVd{font-size:16px;line-height:24px;}.fm06If .ILfuVd.NA6bn,.c2xzTb .ILfuVd.duSGDe,.c2xzTb .ILfuVd.NA6bn.c3biWd{font-size:14px;line-height:1.58;}.c2xzTb .zloOqf{font-size:16px;margin-bottom:16px;margin-top:-8px;}.FLWspc{margin:16px}.LFYdN{font-size:16px;margin:16px}.trNcde .ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:14px;line-height:22px}.HSryR .ILfuVd{line-height:1.5}.ss6qqb .ILfuVd,.ss6qqb .UiGGAb.ILfuVd{font-size:14px;line-height:22px}.c2xzTb .qLYAZd{margin:16px 16px 16px 0;}.d9FyLd{padding:0 0 10px}.hgKElc{padding:0 8px 0 0}.kX21rb{font-size:12px;line-height:1.34;display:inline-block;font-family:Google Sans,arial,sans-serif;padding-right:0;white-space:nowrap}.SPV4pd{display:flex;overflow-x:auto}.SPV4pd::-webkit-scrollbar{display:none}.Vjskue{margin-left:8px}.JT9o0,.ZnIJV{align-items:center;border:1px solid var(–mXZkqc);border-radius:16px;box-sizing:border-box;color:#1a73e8;cursor:pointer;display:flex;height:32px;margin-bottom:16px}.BkHX0d{font-family:Google Sans,arial,sans-serif-medium,sans-serif;font-size:14px;line-height:20px;padding-right:20px;text-align:center;white-space:nowrap}.xv1D5b{display:block;padding:0 8px 0 12px;width:18px;height:18px}.LGOjhe{overflow:hidden;padding-bottom:20px}.trNcde .LGOjhe{overflow:visible}.SALvLe .LGOjhe,.Jb0Zif .LGOjhe,.HSryR .LGOjhe,.IVvPP .LGOjhe{padding:0 0 10px;margin:10px 0 0 0}.s6JM6d .SALvLe .LGOjhe{margin-top:12px;}.s6JM6d .HSryR .LGOjhe{margin-top:12px;padding-bottom:20px}#res .Jb0Zif .qDOt0b h3,.IVvPP .qDOt0b h3{font-size:16px;line-height:1.375}.Jb0Zif .qDOt0b .iUh30,.IVvPP .qDOt0b .iUh30{font-size:14px;line-height:1.58}.IVvPP .qDOt0b{border-bottom:1px solid var(–gS5jXb);padding-bottom:6px}.rjOVwe.ILfuVd{font-size:16px;line-height:20px}.dG2XIf .rjOVwe{padding-top:22px;padding-bottom:0;margin-bottom:1px}.xWd7I{padding-top:12px;}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{font-size:14px;line-height:22px}.n6SJS .ILfuVd{font-size:14px}.ILfuVd.NA6bn,.ILfuVd.duSGDe,.ILfuVd.UiGGAb,.IVvPP .ILfuVd{line-height:20px}.xpd-wa .g-blk{margin:initial}.xpd-wa .dG2XIf{border:initial;box-shadow:none}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd,.w5mTAe .ILfuVd,.W0urI .ILfuVd{font-family:Google Sans,arial,sans-serif;color:var(–YLNNHc)}.fm06If .ILfuVd,.c2xzTb .ILfuVd,.Jb0Zif .c2xzTb .ILfuVd{font-size:20px;line-height:28px}.cUnQKe .ILfuVd,.cUnQKe .UiGGAb.ILfuVd{color:var(–bbQxAb);font-family:Google Sans,arial,sans-serif;font-size:16px;line-height:24px}.fm06If .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b,.c2xzTb .ILfuVd b,.w5mTAe .ILfuVd b,.W0urI .ILfuVd b{background-color:#d3e3fd;color:#040c28;font-weight:500}@media (forced-colors:active){.fm06If .ILfuVd b,.c2xzTb .ILfuVd b,.trNcde .ILfuVd b,.cUnQKe .ILfuVd b{background-color:Mark;color:MarkText}}.IiOSLb .rsGxI.Ww4FFb,.Ww4FFb{background-color:var(–xhUGwc);border-radius:0px;border-width:0px;box-shadow:none}.Ww4FFb .mnr-c:not(:empty),.mnr-c:not(:empty) .Ww4FFb,.Ww4FFb .Ww4FFb{box-shadow:none;margin-bottom:0px}.vt6azd{margin:0px 0px 8px}.vt6azd{margin:0px 0px 30px}.CBvvz{margin:-var(–sds-sys-measurement–space-sm) 0px 0px}.CBvvz{margin:-30px 0px 0px}
Apache Kafka: A Complete Streaming ETL Platform

Kafka Streams API gives applications the stream processing capabilities to transform data, one message or event at a time. These transformations can include joining multiple data sources, filtering data, and aggregating data over a period of time.