Showing posts from 2019

Monitor AWS VPC Connectivity with Python

We recently have the need to cutover our AWS Direct Connects to a different vendor. In order to carry out the change, I was tasked to find a way to monitor Direct Connect connectivities to our on premise network from our hundreds of VPCs in AWS. After some discussion with our network engineers and security team, the solution I end up using is to deploy a single EC2 instance into each those VPCs that has a connection to VGW. We then add those instance IPs into PRTG to monitor with Ping and Http sensors. To allow the instance to be deployed into the targeted AWS accounts, we use CloudFormation StackSet to push out a role into each of those accounts first. The role then allows the "Master Account" to have permission to create and update necessary resources within the target accounts. The instance uses a t3.nano tier alogn with Amazon hvm Linux2 AMI (Use our own hardened AMI in my case). A simple Nginx test page is installed on the instance to allow us to monitor TCP traf

RDP to EC2 with SSM Port Forwarding

Say you have a bunch of Windows servers hosted in AWS. The VPC they are in does not have VPN or Direct Connect connect back to your on premse network. Expose RDP port through public IP for these Windows servers is a very good way to get hacked. So how can we securely connect to the servers in this kind setup? Fortunately we have SSM for the rescue. In August, AWS announced a new feature for SSM Session Manager, which allows us to securely create tunnels between your EC2 instances deployed in private subnets and your local machine. You can read about the announcement here . Here are the steps you can setup for Windows Instances. 1. Configure the Windows EC2 as Managed Instances in SSM. This mainly involves assign a IAM EC2 Role to the instance with SSM policies. Since the focus of this post is about Session Manager Port Forwarding, I won't expand this too much. you can find more details about initial setup of SSM  here . 2. For your existing Windows EC2s, you will need to up

VMware Site Recovery Manager Multi-Site Pair Deployment

I was recently involved in a data center migration project, which used VMware SRM (Site Recovery Manager) as the migration tool to move virtual machines between 3 DCs. The diagram below shows how the setup looks like. The version of SRM is 8.1. [SiteA] <----> [SiteB] <----> [SiteC] VMware documentation refer the above scenario as Shared Recovery Site. For each site-pair, you will need to deploy individual SRM server to ensure the SRM Plug-in ID is unique to that pair. So in our case, SiteB-C pair requires a new SRM server in Site B and Site C. Install SRM The steps below shows the configuration details of the installation process on each SRM server. 1. Provide the PSC server details of the site 2. Provide the vCenter server details of the site 3. Provide the local SRM extension details 4. This is the most important part of the configuration. Make sure you choose Custom Site Recovery Manager Plug-in Identifier . The Plug-in ID needs to be the same on both Site

How Secure is RDP?

Hands up if you have following setup/practices in your organization: A RDP server (Terminal server) that everyone can jump onto. Apart from the IT admins, some users have local admin rights on the box, just so they can run or configure a particular application. To help troubleshooting an issue, your IT admins often RDP to servers directly from user's laptop, which the user is a local admin. A group of users have local admin rights on a particular Windows box, and your Domain Admins also need to RDP to from time to time. If any of above scenario applies to your organization, you might want to consider introducing some changes. This is why: It is well known that Windows Remote Desktop Service has this feature that allows you to connect to another user's session. But obviously you will need to know the other user's credential to do that, right? No, not necessarily. With NT AUTHORITY/SYSTEM account, you can hijack the user's RDP session without the nee

Use Terraform to build server in VMware

Like Cloud Formation and ARM Templates, Terraform enables the way of Infrastructure as Code to provision resources in Clouds, but it also works with on premise infrastructures like VMware vSphere and NSX. I recently have been working on the automation of on premise server provision process. The goal is to provision a Ubuntu server on our vSphere 6.5 environment with iPerf3 installed and configured. It surprises me that there aren’t many useful resources/examples out there when comes to using Terraform with VMware. Yes, there are tons of blog posts about how to build a VMware VM with Terraform, but almost all of those are just touched on very very basic stuff. I can’t find any good reference for how to install and configuration applications within the VM. Without those, I seriously doubt the value of using IaaC. A simple VMware Template with ClickOps will be way more efficient. I did end up finding a very detailed repo about using Terraform to build a Kubernate cluster in VMware in

Sydney AWS Summit 2019 - My Experience

The past week I attended AWS Summit Sydney for three days. It's such an action packed show. Full of brilliant speakers and tons of interesting workshops. I feel so hard to decide on my agenda. In the end, based on the technologies I am interested in and the relevance to my job these are sessions I went for. AWS Innovation Day Keynote - I was late for it. But still It was good to hear the story of Qantas and learn about how they improve performance and efficiency with AWS. I do have doubts of their ambitious goal about flying customers directly from Australia to US and Europa without a stop. Not sure if I want to stuck in a plane for 20 hours... Maybe it's time for a new Concord!   After Keynote, I went around the "Cloud Zone" to talk to different vendors. Among all the vendors, the one left the best impression with me is CloudHealth from VMware. The ability to provide deep analysis of the current spending and indicate detailed remediation plans are lackin

Setup Cross Account S3 Access for Cloudberry Drive

I recently run into a scenario, which one of EC2 instances in our production AWS account (IT) need to access a S3 bucket hosted in a separate account (Marketing). The EC2 instance is a Windows 2008 R2 server. It runs Cloudberry Drive to map the S3 bucket as a local volume for a local application to retrieve the data off it. The easiest way to make this work is to create an IAM user in the and assign it with Access keys. But this is against AWS IAM best practice . Cloudberry Drive does provide the option to use Role for S3 bucket access. Though their documentation is a bit lacking on how to setup this in a cross account scenario. After some Googling, it turns out to be fairly straight forward. Here's how I did it. Let's start with a picture as it helps to clarify where things are in this two account setup. First, in the IT account (111111111111) : 1. Create an IAM role market-s3-role with the following policy: fullaccess-marketing-bucket . The policy allows access to t

The Un-documented Way to Setup AWS SSO with Okta

In this article I would like to share an un-documented way of setting up AWS SSO by using Okta.In case you don’t know what Okta is. It is one of the popular identity management solutions out in the market. It provides Identity as a service through its Web portal and APIs. There is a detailed document provided by Okta walks through steps of how to setting up SAML SSO between your AWS accounts and Okta. So why don’t we follow that? Well, the solution suggested by the official document requires a privileged USER Account to be setup in your Master AWS Account . The account will then be configured to be able to assume roles in all your other accounts. In other words, if this account is compromised, your whole organization’s AWS Accounts are exposed to potential security breach. Ok, what’s the better way then? I actually talked about this in one of my old post Setup SSO Access to AWS Console with Azure AD . Like the Azure AD solution, we will map user groups to different AWS Roles. With

Infrastructure as Code with CloudFormation

Recently I was working on a server migration task, which is to move a Windows IIS web server to AWS. The server’s sole purpose is to redirect bunch of the short URLs to some of the most frequently used long URLs. E.g. if user type in “o365/“ in browser, it will be redirected to . Instead of uplifting the whole Windows server to AWS, I have decided to use a Linux server with Apache to replace this box. Other options like ALB, S3 HTTP Redirection and Route 53 were considered. But none of them can redirect short hostname like “o365”. I have also thought about the idea of using container and serverless options, but given all we need is a single redirect service and the a fixed IP is needed for alias lookup, they are not suitable in this case. The Linux server is provisioned through a CloudFormation template. This allows automation of the whole deploy process. Furthermore, any future updates to the Apache service can also be carried out with an update of the Clo