Ĭurrent consuming offset of the topic partitionĪll metrics of Kafka consumer are also registered under group KafkaSourceReader.KafkaConsumer.įor example, Kafka consumer metric “records-consumed-total” will be reported in metric: topic.partition.1.currentOffset ,Īnd number of successful commits will be reported in metric: In the group .įor example, current consuming offset of topic “my-topic” and partition 1 will be reported in metric: Metrics related to a specific topic partition will be registered Kafka source exposes metrics in Flink’s metric group for monitoring and diagnosing.Īll metrics of Kafka source reader are registered under group KafkaSourceReader, which is aĬhild group of operator metric group. Is only for exposing the progress of consumer and consuming group for monitoring. Note that Kafka source does NOT rely on committed offsets for fault tolerance. Offset committing logic, configured by and in If checkpointing is not enabled, Kafka source relies on Kafka consumer’s internal automatic periodic Kafka source commits the current consuming offset when checkpoints are completed, forĮnsuring the consistency between Flink’s checkpoint state and committed offsets on Kafka brokers.
Partition is considered “idle” and will not hold back the progress of watermarks in downstream operators.ĭescribes details about how to define a WatermarkStrategy#withIdleness. If no records flow in a partition of a stream for that amount of time, then that You will either need to lower the parallelism or add an idle timeout to the The Kafka Source does not go automatically in an idle state if the parallelism is higher than the fromSource ( kafkaSource, new CustomWatermarkStrategy (), "Kafka Source With Custom Watermark Strategy" )ĭetails about how to define a WatermarkStrategy.
The code snippet below shows configuring KafkaConsumer to use “PLAIN” as SASL mechanism and provideĮnv. SetBounded(OffsetsInitializer) has been invoked is overridden by OffsetsInitializer#getAutoOffsetResetStrategy().serializer is always set to ByteArrayDeserializer.key.deserializer is always set to ByteArrayDeserializer.Please note that the following keys will be overridden by the builder even if .checkpoint specifies whether to commit consuming offsets to Kafka brokers on checkpointįor configurations of KafkaConsumer, you can refer to.
Kafka Consumers Offset Committing Behaviour Configuration.Kafka Consumers Topic and Partition Discovery.Kafka Consumers Start Position Configuration.