Tag: devops

Secure SSH Access Using 1Password SSH Agent Vault – No Local Key Storage Required!

Ravi KattaLeave a comment

Why Do We Need This?

In many of our teams, users, including developers, DevOps engineers, and other technical staff, require SSH access to multiple machines. Traditionally, SSH keys have been shared among users, which introduces security risks, as keys can be lost, mismanaged, or exposed. Additionally, assigning and revoking keys when team members join or leave the team is a labor-intensive task.

Possible solution we looked at –

We needed a centralized and secure approach where SSH keys are stored in a vault, rather than being distributed across individual local machines. Users are granted read-only access to the vault, allowing them to authenticate without downloading or manually managing SSH keys.

Tool We found – 1Password SSH Agent

After evaluating different solutions, we found 1Password SSH Agent to be the best fit. With this setup, users can log in seamlessly using:

ssh user@hostname

No need to download , specify or handle keys manually—the 1Password agent securely manages authentication in the background. Let's explore how this works on a windows machine

To use the 1Password SSH agent on Windows, you need to configure 1Password’s built-in SSH agent feature, which allows you to securely manage and use your SSH keys stored in 1Password. Here’s how to set up and use the 1Password SSH agent on Windows:

Steps to Use the 1Password SSH Agent on Windows:

1. Install and Set Up 1Password on Your System

  • If you haven’t already installed 1Password on your Windows machine, download it from the 1Password website.
  • Sign in to your 1Password account.

2. Enable SSH Agent Feature in 1Password

1Password includes an SSH agent that allows you to securely store and use SSH keys directly from your 1Password vault.

  • Open the 1Password app.
  • Go to Settings.
  • Navigate to the Developer section.
  • Toggle on the option for “Use 1Password as your SSH Agent”.

This will configure 1Password to act as an SSH agent for your system.

3. Add SSH Keys to 1Password

To use SSH keys with the 1Password agent, you’ll need to store your private SSH keys in 1Password.

  • In 1Password, click + New Item and choose SSH Key from the list of options.
  • Add your private SSH key (typically from a file like id_rsa), and you can also set a label or other metadata for the key.
  • Save the key.

This key is now stored in your 1Password vault and will be used by the 1Password SSH agent.

4. Configure SSH to Use the 1Password Agent

Next, you need to configure your SSH client to use the 1Password SSH agent. The 1Password SSH agent runs on a named pipe that acts similarly to the ssh-agent provided by OpenSSH.

  • Open your SSH configuration file:
    • Path: C:\Users\YourUsername\.ssh\config (create this file if it doesn’t exist).
  • Add the following configuration to ensure the SSH client uses the 1Password agent:
Host *
    IdentityAgent \\.\pipe\openssh-ssh-agent

This configures the SSH client to use the 1Password agent for all hosts (Host * applies to all SSH connections).

5. Test SSH Authentication with 1Password

Once the 1Password SSH agent is enabled and the configuration is in place, test it by connecting to an SSH server.

For example:

ssh user@hostname

The 1Password SSH agent should now provide the stored SSH key for authentication. If you have multiple SSH keys in 1Password, the agent will offer the appropriate key for the host you’re connecting to.

6. Using 1Password CLI (Optional)

If you prefer more control or want to use 1Password’s CLI tool, you can install the 1Password command-line tool and manage SSH keys and other credentials directly from your terminal.

To install the 1Password CLI on Windows, use PowerShell:

Invoke-RestMethod -Uri https://downloads.1password.com/cli/1password-cli-win64.zip -OutFile 1password-cli-win64.zip
Expand-Archive 1password-cli-win64.zip -DestinationPath C:\op

You can then run op commands to interact with your 1Password vault. However, the built-in SSH agent in the desktop app is usually sufficient for SSH key management.

Key Features of 1Password SSH Agent:

  • Automatic Key Usage: 1Password automatically offers the correct SSH key when authenticating with an SSH server, without needing manual ssh-add commands.
  • Enhanced Security: SSH keys are stored securely in your 1Password vault, protected by your master password and 2FA.
  • Cross-platform: The 1Password SSH agent works across multiple operating systems, including Windows, macOS, and Linux.

Summary

By enabling the 1Password SSH agent and configuring SSH to use it via the IdentityAgent option, you can seamlessly authenticate SSH connections on Windows using the SSH keys securely stored in your 1Password vault. This allows you to centralize SSH key management and take advantage of 1Password’s security features.

AWS Blue Green Deployment

Ravi KattaLeave a comment

A blue/green deployment is a deployment strategy in which you create two separate, but identical environments. One environment (blue) is running the current application version and one environment (green) is running the new application version.

With one of our current environments, the way we upgrade the DB is self-managed and this is tedious and takes a reasonable amount of downtime, this is not acceptable for certain clients (Eg xxx cannot afford more outage).  Apart from downtime, there is a chance for manual errors, as this is self-managed. This problem can be avoided if we have an automated blue green deployment, where it can greatly eliminate the duration of downtime and avoid any manual errors, also if any unexpected error happens with the updated version, say green, we can immediately rollback to the last working version blue.  

With AWS RDS Blue Green deployment, this complete deployment process is  automated with an average downtime of 2 seconds ( downtime can increase based on the amount of data to be replicated and live production traffic, more inputs provided under replica lag section below  ). This can provide an uninterrupted service to the clients with a very minimum outage. 

Let’s  consider the following AWS RDS Deployed in a Single AZ and upgrade to an higher version using  AWS  Blue Green deployment.

Amazon RDS Blue Green Deployment 

By using Amazon RDS Blue/Green Deployments, we  can create a blue/green deployment for managed database changes. A blue/green deployment creates a staging environment that copies the production environment. In a blue/green deployment, the blue environment is the current production environment. The green environment is the staging environment. The staging environment stays in sync with the current production environment using logical replication.

AWS Blue Green deployment helps in all the following use cases. 

  • Major/ Minor version upgrades
  • Schema changes
  • Instance scaling
  • Maintenance updates
  • Engine parameter changes

Blue Green deployment is a Two step process 

  • Step – 1 Create a Blue/Green environment.
  • Step- 2 Switchover to Green environment. 

Step 1 – Create  Blue/Green environment 

  • Select DB( refereed as Blue environment ) which needs to be upgraded.  
  • Go to actions – Create Blue/ Green Deployment. 
  • Provide the following basic configuration
    • Name of the staging environment. 
    • Version of DB to be upgraded. 
    • DB parameter group ( same as prod db parameter group, fine tuning the parameters can reduce the replica lag , more explained in the section Replica Lag below )
  • once submitted, the above configuration creates a copy of the current production environment with a logical data replication. 
  • Traffic will still be flowing through blue environment and data is replicated to Green environment.   


Blue/ Green Deployment RDS configuration 

  • Blue label – Current production environment.
  • Green label – Target Environment.
  • Staging – Logical representation of blue green deployment ( not  a DB instance on its own )

Step 2 – Switch Over

  • Once the Green environment is tested and validated,  the next step is to switch over to the Green environment.
  • Select staging and go to Actions and select the option Switch Over.  
  • Once Switchover is selected, it goes through different steps for switchover
    • Typically it takes one minutes to switch over. 
    • Constantly monitors the health of both blue and green. 
    • Configurable RTO is provided – which rolls back the entire migration if there is any issue with migration , this helps to fail safely. 
    • During this switch over,  we can still have read operations. 
    • Blocks write on blue  and allow green to catch up and then a final switchover is done towards green. 
  • At the end,  once the switchover is done, the entire traffic is redirected to Green environment.  
  • We don’t need any change on the client code, the endpoint remains the same and clients will start  interacting back with the new production green environment as is. 
  • In the interest of safety , the old DB which is a blue environment is not deleted and renamed to -old , we need to manually delete the blue environment or we can keep it as a backup for any further validations.  

Final State after Switchover 

  • Blue DB which is current production environment is renamed to prod-old 
  • Green DB is renamed to prod , which will be the same name as  your current production environment. 

 Multi AZ  RDS Blue green Deployment 

The same entire above flow with Blue Green deployment is validated with Multi AZ  RDS instances and following is the glimpse of the Blue Green environments deployed in Multi AZ. 

Down Time 

  • In a  Single AZ deployment , switch over took 48 seconds  with a test DB. .  
  • In a Multi AZ , switch over took 68 seconds 
  • During this down time any write operations are blocked  and  will result in error – “The MySQL server is running with the –read-only option so it cannot execute this statement” , read operations can still be resumed. 

Replica Lag 

Limitations

  • Available on Amazon RDS for MySQl versions 5.7 and higher.
  • Cross-Region read replicas
  • Amazon RDS Proxy
  • The resources in the blue environment and green environment must be in the same AWS account.
  • More – Blue Green Deployment Limitations

Considerations 

  • During testing, it’s  recommended to keep your databases in the green environment read only. It’s recommended that you enable write operations on the green environment with caution because they can result in replication conflicts in the green environment. They can also result in unintended data in the production databases after switchover.
  • The switchover results in downtime. The downtime is usually under one minute, but it can be longer depending on your workload.
  • The name (instance ID) of a resource changes when you switch over a blue/green deployment, but each resource keeps the same resource ID. For example, a DB instance identifier might be mydb in the blue environment. After switchover, the same DB instance might be renamed to mydb-old1. However, the resource ID of the DB instance doesn’t change during switchover. So, when the green resources are promoted to be the new production resources, their resource IDs don’t match the blue resource IDs that were previously in production
  • When using a blue/green deployment to implement schema changes, make only replication-compatible changes For example, you can add new columns at the end of a table, create indexes, or drop indexes without disrupting replication from the blue deployment to the green deployment. However, schema changes, such as renaming columns or renaming tables, break binary log replication to the green deployment. (https://dev.mysql.com/doc/refman/8.0/en/replication-features-differing-tables.html )

Post Activity : 

  • Pay attention to  channel for any alerts
  • Check Dashboard for any abnormal Activities 
  • Check alerts for error logs
  • check if sudden burst of non-200s
  • look at certain DB metrics 
  • Make sure all the DB parameters are reset back to original values.