Multi-cloud :Keyless authentication : GCP workloads from AWS
Multi-cloud :Keyless authentication (WIF- workload identity federation ): GCP workloads from AWS
Now any enterprise is looking solution on multi and
hybrid cloud which creates and challenges to cloud provider to
implementation service sharing with high security .
Scenario likes
- Date saved in GCP and consumed by AWS
using Lamda function or EKS services
- Data pipelines can be triggered any any
service in AWs or Azure .
service are managed and access in GCP using
service account is a special kind of account typically used by an
application or compute workload, such as a Compute Engine instance, rather than
a person. A service account is identified by its email address, which is unique
to the account.
Service accounts overview | IAM Documentation
| Google Cloud
Service Keys looks like below
But it has some dis-advantages ( mostly on security )
- Keys can easily be leaked
- It's challenging to track key ownership
- Necessary key rotation can be complex
- Management and automation can be
operationally burdensome
Workload identity federation :
Traditionally, applications running outside Google
Cloud can use service account keys to access Google Cloud resources. However,
service account keys are powerful credentials, and can present a security risk
if they are not managed correctly. With identity federation, you can use
Identity and Access Management (IAM) to grant external identities IAM roles,
including the ability to impersonate service accounts. This approach eliminates
the maintenance and security burden associated with service account keys.
A workload identity pool provider is an entity that describes a relationship between Google Cloud and your IdP, including the following:
- AWS
- Azure Active Directory
- On-premises Active Directory Federation
Services (AD FS)
- Okta
- Kubernetes clusters
Workload identity federation follows the OAuth 2.0 token exchange specification. You provide a credential from your IdP to the Security Token Service, which verifies the identity on the credential, and then returns a federated token in exchange.
Here , providing step by steps accessing BQ API from AWS EC2 instance using WIF [ workload identity federation ]
Steps would be as follows
1. Create AWS role for EC2 instance.
2. Create EC2 instance from console / terraform attached
with same created role
3. Create WIF in GCP
4. Create identity pool provider in GCP for your AWS account ID
5. Create a service account with BQ role and WIF.
6. Generate google credential.
7. Attached and use the same credential json file with
EC2
gcloud iam workload-identity-pools create awspool \
--location="global" \
--description="datapath example" \
--display-name="aws pool"
gcloud iam workload-identity-pools providers
create-aws awsdata \
--workload-identity-pool="awspool" \
--account-id="471264607017" \
--location="global"
Note : You can have multiple providers in same service pool
check it from Console:
STEP 5: Create service account with WIF and BQ role
Allow only specific AWS role to give access to GCP service account
We want AWS
to access some things in Google Cloud. To do this, we need to create something
called a service account. This service account acts a bit like a key, unlocking
certain areas in Google Cloud for AWS to use.
We also
need to give this service account a special role. We call this role
'roles/iam.workloadIdentityUser'. With this role, whenever an AWS service wants
to use something in Google Cloud, it can act as if it were this service
account. This means it can do whatever this service account is allowed to do.
Now, we
have a choice. We can decide who gets to use this service account. We might
let:
Just
one specific user from AWS
1.
Any user that has a specific
role in AWS (which is what we're doing here )
2.
Any user that has a specific
characteristic or value
3. Any user in a group (or 'pool') of users
gcloud iam service-accounts add-iam-policy-binding awstobg@subhratestproject.iam.gserviceaccount.com \
--role=roles/iam.workloadIdentityUser \
--member="principalSet://iam.googleapis.com/projects/773595548939/locations/global/workloadIdentityPools/awspool/attribute.aws_role/arn:aws:sts::471264607017:assumed-role/awsec2"
awstobg@subhratestproject.iam.gserviceaccount.com :: GCP Service account
773595548939 :: GCP project number
471264607017 :: AWS project ID
aswpool :: workload identity pool id
awsec2 :: AWS role which we created
STEP 6 : Generate credential
Now we are
going to create credential json key whcih will be used for
AWS account 471264607017 and role awsec2 .so if anyone wants to
use it with any AWS service from another account or other role in the same
account it will not work. So no issue on key security
now
gcloud iam workload-identity-pools create-cred-config \
projects/773595548939/locations/global/workloadIdentityPools/awspool/providers/awsdata \
--service-account=awstobg@subhratestproject.iam.gserviceaccount.com \
--output-file=configoutput.json \
--aws
Testing from AWS
Now we have
to test the BQ connectivity and service from AWS EC@ machine we have created
Step2 :
-
Log in AWS console and open same
EC2 instance
-
Install Google and Big query library to use GCP BQ API from AWS EC2 instance
pip install google-auth
pip install —
upgrade google-cloud-bigquery
-
Upload the same configoutput.json
in EC2 instance
-
Create python file and execute
the same to check
ubuntu@ip-172-31-9-46:~$ cat awstobg.py
import os
from google.cloud import bigquery
# Set environment variables
os.environ["GOOGLE_CLOUD_PROJECT"] = 'subhratestproject' # The GCP project ID
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = 'configoutput.json' # Configuration file
# Create BigQuery client object
client = bigquery.Client()
query = """
SELECT trip_id,subscriber_type
FROM `bigquery-public-data.austin_bikeshare.bikeshare_trips` LIMIT 10 """
# Run the query
query_job = client.query(query)
print("Query result:")
for row in query_job:
print(f"subscriber={row['subscriber_type']}")
Comments
Post a Comment