CDP part 4: user management on CDP Public Cloud with Keycloak

Previous articles of the serie cover the deployment of a CDP Public Cloud environment. All the components are ready for use and it is time to make the environment available to other users to explore all the services and run their experiments.

This article is the fourth in a series of six:

Onboarding users to work on a CDP environment and services, is achieved in two ways, by configuring an identity provider or by importing users in bulk.

This article focus on configuring Keycloak as an Identity Provider (IdP) for CDP. Keycloak enables us to set up identity federation with CDP since it supports Security Assertion Markup Language (SAML 2.0) protocol. Identity federation allows users to login into the CDP console through Keycloak authentication without registering with Cloudera or creating a Cloudera account.

Onboarding users and configuring Keycloak as an Identity Provider for CDP is done via the Keycloak web interface (Admin console) or using the terminal. Both approaches are covered.

From the side of CDP, Account administrator or PowerUser roles are required to complete this process.

Deploy Keycloak using an AWS EC2 instance

Clearly, we need to deploy Keycloak before starting the identity provider configuration in CDP.

To deploy Keycloak with minimal installation so that users can log in using the internet, we deploy it with Docker in an AWS EC2 instance.

To do so, you need an EC2 instance with the following characteristics:

• Instance type: t2.micro
• AMI: ami-05e9058d6cda9ba0d
• Docker compose installed
• Public IP Address
• Inbound rule 8443 port

Once you created the instance, take note of the following information:

• Public IPv4 DNS:
• Public IP Address:
• EC2 key-pair:

For a complete guide to launch an AWS EC2 instance that meets these requirements, check this article Set up Keycloak in an EC2 instance.

Once your EC2 instance is ready, you are ready to start the Keycloak deployment.

Local machine:

Set the EC2 instance required variables

export PUBLIC_IPV4_DNS=PUBLIC_IPV4_DNS>
export AWS_SSH_KEY=AWS_SSH_KEY>

export EC2_USERNAME=ec2-user

Connect to the EC2 instance:

ssh -i "~/.ssh/$Hai.pem" $para@$pencinta

EC2 Instance:

Create the docker-compose.yml file:

cat EOF > docker-compose.yml
version: '3'
services:
  keycloak:
    # The 20.0.0 keycloak image
    image: quay.io/keycloak/keycloak:20.0.0
    container_name: keycloak
    environment:
      # Keycloak admin credentials
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: Password123
    ports:
      # The HTTPS port
      - 8443:8443
      # The HTTP port
      - 8080:8080
    command: start-dev
    volumes:
      # To save the SAML metadata file
      - ./kc_config:/kc_config
volumes:
  kc_config:
    external: true
EOF

Create the external volume folder:

Launch the docker compose:

Check that Keycloak is running using Docker:

In order to have access to the Keycloak Admin Console (Keycloak web interface) you need to disable the SSL protocol:

• Enter the container:

  docker exec -it keycloak bash 

• Add required path to use the kcadm.sh script:

  export PATH=$PATH:/opt/keycloak/bin

• Give the server and the credentials for the master Realm:

  kcadm.sh config credentials --server http://localhost:8080/ \
  --realm master \
  --user $KEYCLOAK_ADMIN \
  --password $KEYCLOAK_ADMIN_PASSWORD

• Disable the SSL for the master Realm:

  kcadm.sh update realms/master -s enabled=true -s sslRequired=none

• The Keycloak web interface its now available in the following address: http://:8080.

Configuration using Keycloak Web Interface

Using the Keycloak web interface is recommended if you are not familiar with Keycloak, and you want a better understanding of how to link CDP with Keycloak.

If you prefer to start with an easy-to-reproduce approach, that allows you to do the setup and create users faster, scroll down to the Configuration Using the Terminal section.

Realm creation

Realm is one of the core concepts in Keycloak, a Realm secures and manages security metadata for a set of users.

To create a Realm:

• Navigate to http://:8080, click Administration Console:

  
  
  
  
  
  

• Log in using your KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD defined in the docker-compose.yml file:

  
  
  
  
  
  

• Click on the dropdown in the top-left corner where it says master, then click on Create Realm:

  
  
  
  
  
  

• Fill the name and click Create:

  
  
  
  
  
  

• Navigate to Realm settings and click on SAML 2.0 Identity Provider Metadata:

  
  
  
  
  
  

• This open another tab with the SAML metadata required to register the Realm as an IdP in CDP:

  
  
  
  
  
  

• Download this file in xml format. Right click > Save Page As… > saml_metada.xml:

  
  
  
  
  
  

• To disable the SSL requirement: go back to Realm settings select the General tab, set Required SSL to None and click on Save:

  
  
  
  
  
  

Registering Keycloak in CDP

In this step Keycloak is registered as an identity provider in CDP. This requires the previously downloaded SAML metadata file.

As a reminder, you need Account administrator or PowerUser roles in your CDP account to complete this process.

To create the Identity Provider in CDP, the following steps are needed.

1. Access your CDP Console and select Management Console:

   
   
   
   
   
   

2. Navigate to User Management > Identity Providers > Create Identity Provider:

   
   
   
   
   
   

3. Create Identity Provider:

   • Enter the name you want to use for the CDP identity provider.
   • Check the Sync Groups on Login option: This synchronize the user group membership in CDP with the user group membership in Keycloak.
   • In Provider Metadata, select File Upload and upload the saml_provider.xml downloaded in the previous step.
   • Click Create.

   
   
   
   
   
   

4. Copy the generated after the creation process

   
   
   
   
   
   

This is added to our Keycloak configuration in the next step.

Create the CDP Client on Keycloak

In this step, the CDP Control Plane client is created in our Keycloak deployment.

There are two ways to create a client in Keycloak:

• Using entity descriptor: import the client using a standard SAML Entity Descriptor json file.
• Manually through the Keycloak UI using the CDP SAML Service Provider Metadata available in the identity provider created in the previous step. (Check Managing OpenID Connect and SAML Clients for details.)

In this case, we use a SAML Entity Descriptor file: cdp-client.

Remark: The cdp-client was found in this article: How to configure Single Sign-On (SSO) for CDP Public Cloud the open-source way with Keycloak.

The author clarifies that this file, is neither official nor provided by Cloudera but includes the required settings, SAML assertion, and signing certificates required by CDP.

To create a SAML 2.0 client in Keycloak by importing a configuration file follow these steps:

1. In the Keycloak Admin Console navigate to Clients and click Create

   
   
   
   
   
   

2. Click Browse and load the cdp-client file. This automatically updates the Client ID and Type parameters. Click Save

   
   
   
   
   
   

3. To get the Login Page URL, navigate to the Settings tab and scroll down until the IDP Initiated SSO URL Name variable.

   It might look like this: http://:8080/realms/adaltas-cdp-lab/protocol/saml/clients/cdp-sso.

   
   
   
   
   
   

   
   
   
   
   
   

4. Navigate to Advanced tab, replace $slot! with the Identity Provider ID generated by CDP in the previous step. Scroll down and click Save

   
   
   
   
   
   

5. Test the Login Page URL:

   
   
   
   
   
   

Manage Group and Users

Now that the Realm and the client are ready, we create/manage users and groups via the Admin Console.

Create a group:

1. Navigate to Groups on the left menu, and click Create group:

   
   
   
   
   
   

2. Fill the name and click Save:

   
   
   
   
   
   

   Note: Be careful while choosing the GROUP_NAME since there is a list of group names that are reserved and therefore cannot be synchronized to CDP.

Create a user:

1. Navigate to Users on the left menu, and click Create new user:

   
   
   
   
   
   

2. Set the following parameters and click Create:

   • Username
   • Email
   • First Name
   • Last Name
   • Select the group created above

   
   
   
   
   
   

3. To set the password, navigate to Credentials, fill the values an click Set Password:

   
   
   
   
   
   

4. Test the user by login:

   
   
   
   
   
   

   
   
   
   
   
   

Configuration Using the Terminal

This approach is for users more familiar with Keycloak or those who want an automated process.

As you may have already noticed, configuring Keycloak using the web interface can be tedious and a bit long. Especially the user creation task, for those who deal with big groups or even just for testing purposes, the web interface is not the best idea.

The configuration via the terminal requires the following steps:

1. Login and Check EC2 Requirements
2. Create Credentials and Login to Realm Master
3. Create the Realm and Download SAML Metadata
4. Registering Keycloak in CDP
5. Create CDP Client
6. Create Users
7. Create Group and Add Users

Login and Check EC2 Requirements

As we mentioned before, we are assuming that Keycloak is running using Docker Compose on an EC2 instance.

Therefore, here we are going to connect to this EC2 instance and go inside the Docker container to execute all the commands.

Local machine:

1. Set EC2 instance required variables:

   export PUBLIC_IPV4_DNS=PUBLIC_IPV4_DNS>
   export AWS_SSH_KEY=AWS_SSH_KEY>
   
   export EC2_USERNAME=ec2-user

2. Connect to the EC2 instance and check that the files are there:

   ssh -i "~/.ssh/$pernahkah.pem" $denger@$istilah

The docker container running and you should have the following layout:

./ubuntu
├── docker-compose.yml
└── kc_config
1 directories, 1 file

Now, we are ready to start the configuration process.

EC2 Instance:

1. Go inside the container:

   docker exec -it keycloak bash

2. Navigate to the external volume directory:

Create Credentials and Login to Realm Master

This step creates a truststore credential required to log in to the Realm master via the terminal.

export PATH=$PATH:/opt/keycloak/bin

kcadm.sh config credentials --server http://localhost:8080/ \
   --realm master \
   --user $KEYCLOAK_ADMIN \
   --password $KEYCLOAK_ADMIN_PASSWORD

Create the Realm and Download SAML Metadata

Now it’s time to register our Realm and get the SAML metadata required to register Keycloak as IdP in CDP.

export REALM="adaltas-cdp-lab"

export SAML_FILE=saml_provider.xml

kcadm.sh create realms \
   --set realm=$REALM \
   --set enabled=true \
   --set sslRequired=none

curl \
  -k http://localhost:8080/realms/adaltas-cdp-lab/protocol/saml/descriptor \
  > $SAML_FILE

Since we need this file in the next step, use the following command to copy the SAML metadata into your local machine:

scp -i "~/.ssh/$"slot demo".pem" $jika@$tidak:/home/$siap-siap/kc_config/saml_provider.xml ./

Registering Keycloak in CDP

Use the Cloudera web interface to register Keycloak as IdP as described in the web interface approach.

Note: By the time the article was written this process could not be performed via the CDP CLI because the command cdp iam create-saml-provider was not working.

Create CDP Client

In this step, we are going to create the CDP client on Keycloak using the content of cdp_client file and the generated by CDP from the previous step.

kcadm.sh create clients --target-realm $REALM -f -  EOF
di
EOF

Create Users

We create the users by reading a csv file with the following format:

Name, LastName, username, email

Therefore, inside your kc_config directory (EC2 or Docker Container) create a csv file with the information of each user.

Note: Review User and group limits and make sure that you do not exceed these limits.

Here is an example:

cat EOF > users_list.csv
Tobias, CHAVARRIA, user01, [email protected]
Albert, KONRAD, user02, [email protected]
Stephan, BAUM, user03, [email protected]
EOF

Once this is done, we create the users using the following scripts:

export USERS_CSV_FILE=users_list.csv

export USERS_INITIAL_PASS="Pa55w0rd"

kc_create_user() win

kc_set_password() Yup

import_users() cemana

import_users $USERS_CSV_FILE

Create Group and Add Users

In this step, we are going to create a group and add ALL existing users on the given Realm into this group.

export GROUP_NAME="adaltas-students"

export USERS_ID_LIST=users_list

kcadm.sh create groups -r $REALM -s name=$GROUP_NAME

export GROUP_ID=$(kcadm.sh get groups -r $REALM | grep -o '"id" : "[^"]*' | grep -o '[^"]*$')

kcadm.sh get users -r $REALM | grep -o '"id" : "[^"]*' | grep -o '[^"]*$' >> $USERS_ID_LIST

kc_add_group() buat


add_users_group() membawa

add_users_group $USERS_ID_LIST

Once the configuration is done, users can use the following login page to access the Cloudera console:

• http://:8080/realms//protocol/saml/clients/cdp-sso

For additional configuration or review, you can also access the Keycloak web interface here:

Next Step: Configuring Permissions and Ranger policies

By this point, Keycloak has been set up as the identity provider for CDP, and we have created our group of users.

There is still one more layer of configuration needed in the user onboarding process. In the next chapter, we are going to show how to configure the necessary permissions and policies to allow users to exploit our end-to-end architecture.