Which AWS service or feature can be used to create a private connection between an on

AWS PrivateLink allows for one-way secure connection access from your VPC to Confluent Cloud with an added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity.

The following diagram summarizes the AWS PrivateLink architecture with the customer VPC/account and the Confluent Cloud VPC/account.

Prerequisites¶

A Confluent Cloud network (CCN) of type PRIVATELINK in AWS. If a network does not exist, follow the procedure below in Create a Confluent Cloud network in AWS.
To use AWS PrivateLink with Confluent Cloud, your VPC must allow outbound internet connections for DNS resolution, Confluent Cloud Schema Registry, and Confluent CLI.
- DNS requests to the public authority traversing to private hosted zone is required.
- Confluent Cloud Schema Registry is accessible over the internet.
- Provisioning new ksqlDB clusters requires Internet access. After ksqlDB clusters are up and running, they are fully accessible over AWS PrivateLink connections.
- Confluent CLI requires internet access to authenticate with the Confluent Cloud control plane.
Confluent Cloud Console components, like topic management, require additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with AWS PrivateLink, see Access Confluent Cloud Console with Private Networking.

Warning

For limitations of the AWS PrivateLink feature, see Limitations below.

Create a Confluent Cloud network in AWS¶

To create a Dedicated cluster with AWS PrivateLink, you will need to create a Confluent Cloud network first in the required cloud and region.

In the Confluent Cloud Console, go to the Network management page for your environment.
Click Create your first network if this is the first network in your environment, or click + Add Network if your environment has existing networks.
Select AWS as the Cloud Provider and the desired geographic region.
Select the PrivateLink connectivity type and click Continue.
Specify a Network Name, review your configuration, and click Create Network.

In most cases, it takes up to 15 to 20 minutes to create a Confluent Cloud network. Note the Confluent Cloud network ID from the response to specify it in the following commands.

After successfully provisioning the Confluent Cloud network, you can add Dedicated clusters within your Confluent Cloud network by using either of the following procedures:

Confluent Cloud Console: Create a Cluster in Confluent Cloud
Cluster Management API: Create a cluster

Register your AWS account with Confluent Cloud¶

To make an AWS PrivateLink connection to a cluster in Confluent Cloud you must register the AWS Account ID you wish to use. This is a security measure so Confluent can ensure only your organization can initiate AWS PrivateLink connections to the cluster. AWS PrivateLink connections from a VPC not contained in a registered AWS account will not be accepted by Confluent Cloud.

You can register multiple AWS accounts to the same Confluent Cloud cluster, and AWS PrivateLink connections can be made from multiple VPCs in each registered AWS account.

In the Confluent Cloud Console, go to your network resource in the Network Management tab and click + PrivateLink Access.
Enter the 12-digit AWS Account Number for the account containing the VPCs you want to make the AWS PrivateLink connection from.
Note the VPC Endpoint service name to create an AWS PrivateLink connection from your VPC to the Confluent Cloud cluster. This URL will also be provided later.
Click Save.

Your AWS PrivateLink connection status will transition from “Pending” to “Active” in the Confluent Cloud Console. You still need to configure the Private Endpoints in your VPC before you can connect to the cluster.

Create an AWS PrivateLink connection to Confluent Cloud¶

Follow this procedure to create an AWS PrivateLink connection to a Confluent Cloud cluster on AWS using the Confluent Cloud Console or REST APIs.

Set up the VPC Endpoint for AWS PrivateLink in your AWS account¶

After the connection status is “Active” in the Confluent Cloud Console, you must configure Private Endpoints in your VPC from the AWS Management Console to make the AWS Private Link connection to your Confluent Cloud cluster.

Note

Confluent recommends using a Terraform configuration for setting up Private Link endpoints. This configuration automates the manual steps described below.

Prerequisites

In the Confluent Cloud Console, find the following information for your Confluent Cloud cluster under the Cluster Settings section and Confluent Cloud network under Confluent Cloud Network overview.

Kafka Bootstrap (in the General tab)
Availability Zone IDs (in the Networking tab)
VPC Service Endpoint Name (in the Networking tab)
DNS Domain Name (in the Networking tab)
Zonal DNS Subdomain Names (in the Networking tab)

Verify subnet availability.
The Confluent Cloud VPC and cluster is created in specific zones that, for optimal usage, should match the zones of the VPC you want to make the AWS PrivateLink connections from. You must have subnets in your VPC for these zones so that IP addresses can be allocated from them. It is allowed to also have subnets in zones outside of these. AWS Zone IDs should be used for this. You can find the specific Availability Zones for your Confluent Cloud cluster in the Confluent Cloud Console.
Note

Please note: Because Availability Zone names (for example, us-west-2a) are inconsistent across AWS accounts, Availability Zone IDs (like usw2-az1) are used.
Verify that DNS hostnames and DNS resolution are enabled.
1. Open the AWS Management Console and go the the VPC Dashboard at //console.aws.amazon.com/vpc/home.
2. In the navigation menu under VIRTUAL PRIVATE CLOUD, click Your VPCs. The Your VPCs page appears.
3. Select your VPC, click Actions, and Edit DNS hostnames. The Edit DNS hostnames page appears.
4. Verify that DNS hostnames is enabled.
5. Click Actions and then select Edit DNS resolution. The Edit DNS resolution page appears.
6. Verify that DNS resolution is enabled.
Create the VPC endpoint.
1. Open the AWS Management Console and go the the VPC Dashboard at //console.aws.amazon.com/vpc/home.
2. In the navigation menu under VIRTUAL PRIVATE CLOUD, click Endpoints. The Endpoints page appears.
3. Click Create endpoint. The Create endpoint page appears.
4. Under Service category, select Other endpoint services.
5. Under Service settings, enter the Service name for your Confluent Cloud VPC Service Endpoint Name. You can find this in the Confluent Cloud Console.
6. Click Verify service. If you get an error, ensure that your account is allowed to create PrivateLink connections.
7. Under VPC, select the VPC in which to create your endpoint.
8. Click Create endpoint.
  Your VPC endpoint is created and displayed. Copy the VPC Endpoint ID for later use.
9. Note the availability zones for your Confluent Cloud cluster from the Networking tab in the Confluent Cloud Console. Select the service in these zones. Ensure the desired subnet is selected for each zone. Failure to add all zones as displayed in the Confluent Cloud Console can cause connectivity issues to brokers in the omitted zones, which can result in an unusable cluster.
  Note
  Confluent Cloud single availability zone clusters need service and subnet selection in one zone whereas Confluent Cloud multi-availability zone clusters need service and subnet selection in three zones.
10. Select or create a security group for the VPC Endpoints.
  - Add three inbound rules for each of ports 80, 443, and 9092 from your desired source (your VPC CIDR). The Protocol should be TCP for all three rules.
  - Port 80 is not required, but is available as a redirect only to https/443, if desired.
11. Wait for acceptance by Confluent Cloud. This should happen almost immediately (less than a minute). After it is accepted, the endpoint will transition from “Pending” to “Active”.

Set up DNS records to use AWS VPC endpoints¶

DNS changes must be made to ensure connectivity passes through AWS PrivateLink in the supported pattern. Any DNS provider can be used - AWS Route53 (used in this example) is not required. Any DNS provider that can ensure DNS is routed as follows is acceptable.

Important

In the AWS Management Console, the Enable DNS name setting under Additional settings (only appearing after the VPC is selected) must be disabled when creating a private endpoint. By default, this setting is enabled.

Update DNS using AWS Route53 in the AWS Management Console:

Create the Private Hosted Zone.
1. Click Create Hosted Zone.
2. Paste Confluent Cloud DNS into Domain Name. This can be found in the Confluent Cloud Console.
3. Change Type to Private Hosted Zone for Amazon VPC.
4. Select the VPC ID where you added the VPC Endpoint.
5. Click Create.
Set up DNS records for Confluent Cloud single availability zone clusters as follows:
1. Create the following record with the Create Record button using the VPC Endpoint DNS Name map from the previous setup in the form.
  *.$domain CNAME “The lone zonal VPC Endpoint” TTL 60
  For example:
  *.l92v4.us-west-2.aws.confluent.cloud CNAME vpce-09f9f4e9a86682eed-9gxp2f7v-us-west-2c.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60
Set up DNS records for Confluent Cloud multi-availability zone clusters as follows:
1. Create the following records with the Create Record button
  using the VPC Endpoint DNS Name map from the previous setup in the form.
  
  *.$domain CNAME “All Zones VPC Endpoint” TTL 60
  For example:
  *.l92v4.us-west-2.aws.confluent.cloud CNAME vpce-09f9f4e9a86682eed-9gxp2f7v.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60
  The CNAME is used to ensure AWS Route53 health checks are used in the case of AWS outages.
2. Create one record per zone (repeat for all zones) in the form.
  *.$zoneid.$domain CNAME “Zonal VPC Endpoint” TTL 60
  For example:
  *.usw2-az3.l92v4.us-west-2.aws.confluent.cloud. CNAME vpce-09f9f4e9a86682eed-9gxp2f7v-us-west-2a.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60 *.usw2-az2.l92v4.us-west-2.aws.confluent.cloud. CNAME vpce-09f9f4e9a86682eed-9gxp2f7v-us-west-2c.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60 *.usw2-az1.l92v4.us-west-2.aws.confluent.cloud. CNAME vpce-09f9f4e9a86682eed-9gxp2f7v-us-west-2b.vpce-svc-04689782e9d70ee9e.us-west-2.vpce.amazonaws.com TTL 60

Validate connectivity to Confluent Cloud¶

From an instance within the VPC (or anywhere the previous step’s DNS is set up), run the following to validate Kafka connectivity through AWS PrivateLink is working correctly.
1. Set an environment variable with the cluster bootstrap URL.
  export BOOTSTRAP=$<bootstrap-server-url>
  The Bootstrap URL displayed in Confluent Cloud Console includes the port (9092). The BOOTSTRAP value should include the full hostname, but do not include the port. This is so that you can run the openssl s_client -connect <host>:<port> command with the required values.
  For example:
  export BOOTSTRAP=lkc-nkodz-0l6je.us-west-2.aws.confluent.cloud
2. Test connectivity to your cluster by running the openssl s_client -connect <host>:<port> command, specifying the $BOOTSTRAP environment variable for the <host> value and 9092 for the <port> value.
  openssl s_client -connect $BOOTSTRAP:9092 -servername $BOOTSTRAP -verify_hostname $BOOTSTRAP </dev/null 2>/dev/null | grep -E 'Verify return code|BEGIN CERTIFICATE' | xargs
  To run the openssl s_client -connect command, the -connect option requires that you specify the host and the port number. For details, see the openssl s_client documentation.
3. If the return output is -----BEGIN CERTIFICATE----- Verify return code: 0 (ok), connectivity to the bootstrap is confirmed.
  Note
  You might need to update the network security tools and firewalls to allow connectivity. If you have issues connecting after following these steps, confirm which network security systems your organization uses and whether their configurations need to be changed. If you still have issues, run the debug connectivity script and provide the output to Confluent Support for assistance with your PrivateLink setup.
Next, verify connectivity using the Confluent Cloud CLI.
1. Sign in to Confluent CLI with your Confluent Cloud credentials.
2. List the clusters in your organization.
  confluent kafka cluster list
3. Select the cluster with AWS PrivateLink you wish to test.
  confluent kafka cluster use ...
  For example:
  confluent kafka cluster use lkc-a1b2c
4. Create a cluster API key to authenticate with the cluster.
  confluent api-key create --resource ... --description ...
  For example:
  
  confluent api-key create --resource lkc-a1b2c --description "connectivity test"
5. Select the API key you just created.
  confluent api-key use ... --resource ...
  For example:
  confluent api-key use WQDMCIQWLJDGYR5Q --resource lkc-a1b2c
6. Create a test topic.
  confluent kafka topic create test
7. Start consuming events from the test topic.
  confluent kafka topic consume test
8. Open another terminal tab or window.
9. Start a producer.
  confluent kafka topic produce test
10. Type anything into the produce tab and hit Enter; press Ctrl+D or Ctrl+C to stop the producer.
11. The tab running consume will print what was typed in the tab running produce.

You’re done! The cluster is ready for use.

Limitations¶

AWS PrivateLink is only available for use with Dedicated clusters.
Existing Confluent Cloud networks cannot be converted to use AWS PrivateLink.
Cross-region AWS PrivateLink connections are not supported.
See also: Prerequisites.

Connectors¶

Fully-managed Confluent Cloud connectors can connect to sources or sinks using a public IP. Sources or sinks in the customer network with private IP are not supported.

Availability zones¶

Japan (Osaka) region ap-northeast-3 and Jakarta region ap-southeast-3 are not supported for AWS PrivateLink clusters in Confluent Cloud. For these regions, you can use VPC peering for clusters, or use AWS PrivateLink with clusters in different regions.

All AWS availability zones, except use1-az3, are supported in the us-east-1 region.

Single availability-zone clusters¶

Each Confluent Cloud single zone cluster that uses AWS PrivateLink access is provisioned with service endpoints in one availability zone. The availability zone is selected based on Confluent Cloud placement policies.

To ensure connectivity over AWS PrivateLink connections, provision subnets in your VPC that minimally include the single availability zone in which the AWS PrivateLink access is provisioned.

Which AWS service or feature can be used to create a virtual network that is private and isolated in the AWS cloud?

Amazon VPC enables you to build a virtual network in the AWS cloud - no VPNs, hardware, or physical datacenters required. You can define your own network space, and control how your network and the Amazon EC2 resources inside your network are exposed to the Internet.

Which AWS services or features give users the ability to create a network connection between two VPCs?

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses.

Which AWS service can be used to establish a dedicated private network connection between AWS and your data Centre?

AWS Direct Connect makes it easy to establish a dedicated connection from an on-premises network to one or more VPCs in the same region. Using private VIF on AWS Direct Connect, you can establish private connectivity between AWS and your data center, office, or colocation environment, as shown in the following figure.

Which AWS network feature can help you make a private connection between AWS and your data center?

AWS Direct Connect is a networking service that provides an alternative to using the internet to connect to AWS. Using AWS Direct Connect, data that would have previously been transported over the internet is delivered through a private network connection between your facilities and AWS.