Embedded Files
List all connectors
List all connectors
https://gerardnico.com/dit/kafka/connect/rest#list_connector_plugins
curl -s localhost:8083/connector-plugins | jq .
Adding JDBC Connector
Adding JDBC Connector
curl -d @"connect-file-source.json" \
-X POST http://localhost:8083/connectors \
-H "Content-Type: application/json"
connect-file-source.json
______________
{
"name": "debezium-postgres-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "docker.for.mac.localhost",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname" : "northwind",
"database.server.name": "northwind",
"table.whitelist": "customers",
"tasks.max": 1,
"plugin.name": "pgoutput"
}
}
______________
or
curl -i -X POST http://localhost:8083/connectors \
-H "Content-Type: application/json" \
-d '{ \
"name": "postgres-jdbc-connector2", \
"config": { \
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", \
"connection.url": "jdbc:postgresql:docker.for.mac.localhost:5432/northwind", \
"connection.user": "postgres", \
"connection.password": "postgres", \
"poll.interval.ms" : 3600000, \
"table.whitelist" : "customers", \
"tasks.max": 1, \
"mode": bulk \
} \
}'
Connectors via Kafka Connect UI
Connectors via Kafka Connect UI
name=local-postgres-jdbc-northwind
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:postgresql://docker.for.mac.localhost:5432/northwind
topic.prefix=local_
connection.user=postgres
connection.password=postgres
table.whitelist=customers
mode=bulk
#key.converter=org.apache.kafka.connect.json.JsonConverter
#value.converter=org.apache.kafka.connect.json.JsonConverter
Property Definitions
Property Definitions
Good References
Good References
https://www.baeldung.com/kafka-connectors-guide
https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/
Trouble Shooting Tips
Trouble Shooting Tips
JDBC Driver missing
SSH Login to where Kafka Connect is running.
Find where the plugin is running. In this case kafka-connect-jdbc-N.N.N.jar
find / -name kafka-connect-jdbc\*.jar
Add the jdbc driver i.e. oracle jdbc
Required to restart kafka connect work.
decoderbufs missing.
Edit postgresql.conf and change the wal_level to replication
Solution Details: https://stackoverflow.com/questions/59978213/debezium-could-not-access-file-decoderbufs-using-postgres-11-with-default-plug
Confluent cli
Confluent cli
Install confluent cli in Mac
curl -L --http1.1 https://cnfl.io/cli | sh -s -- -b /usr/local/bin
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
mode=bulk
topic.prefix=dw_test_
connection.password=xxx
tasks.max=1
connection.user=diginex
value.converter=org.apache.kafka.connect.json.JsonConverter
connection.url=jdbc:postgresql://dw-api-rds-dev-1.cq1ivvhcnwjt.ap-southeast-1.rds.amazonaws.com:5432/dw
key.converter=org.apache.kafka.connect.json.JsonConverter
table.whitelist=bank
name=PostgresConnector
connector.class=io.debezium.connector.postgresql.PostgresConnector
database.dbname=dw
database.user=postgres_ro
tasks.max=1
database.hostname=dw-api-rds-dev-1.cq1ivvhcnwjt.ap-southeast-1.rds.amazonaws.com
database.password=V8BZYV-bFDKvaCtsDkj=slfnBvykBRAk)18BKIAd
database.server.name=dwname
database.port=5432
schema.whitelist=bank
connector.class=io.debezium.connector.postgresql.PostgresConnector
database.dbname=accounts
database.hostname=dw-api-rds-dev-1.cq1ivvhcnwjt.ap-southeast-1.rds.amazonaws.com:5432/dw
database.password=V8BZYV-bFDKvaCtsDkj=slfnBvykBRAk)18BKIAd
database.port=5432
database.server.name=porkchop_dbz_topicprefix
database.sslmode=disable
database.user=postgres_ro
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schema.registry.url=http://schema-registry-cp-schema-registry:8081
key.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schema-registry-cp-schema-registry:8081
value.converter=io.confluent.connect.avro.AvroConverter
plugin.name=pgoutput
slot.name=porkchop_dbz_slotname
tasks.max=1
Page updated
Google Sites
Report abuse