How Automated Provisioning Tools Pave the Way to Multi-Cloud Adoption

How Automated Provisioning Tools Pave the Way to Multi-Cloud Adoption

The multi-cloud era has not only started — it’s already enjoying massive adoption with infrastructure as code and provisioning tools at the forefront of the movement. And those trends hold true across regions, industries, and even different levels of cloud spending.

That’s just a taste of the surprising statistics and interesting takeaways from the inaugural HashiCorp State of Cloud Strategy Survey of 3,200+ practitioners and decision-makers from the HashiCorp opt-in contact database.

In this post, we’ll take a closer look at some of the provisioning-related insights revealed in the survey, which explored the specific challenges related to the core workflows of the cloud-adoption journey: provisioning, security, application deployment, and networking.

»Provisioning Is Most Popular Category of Automation Tools

The survey showed an overwhelming share of organizations find infrastructure automation tools are critical to overcoming their cloud challenges. And provisioning was the most widely used category of infrastructure automation tooling. Roughly 90% of respondents indicated they’re either already using automated provisioning tools or will be in the next 12 months for provisioning purposes — only about 3% have no plans to use them.

Not surprisingly, perhaps, usage of automated provisioning tools was correlated with cloud spending. Approximately 65% of companies budgeting less than $100,000 in annual cloud spending already use automated provisioning tools, compared to 79% of organizations with annual cloud budgets of $2 million or more.

Regionally, it turns out that respondents in Europe/Middle East/Africa report the greatest use of provisioning automation tools (78%). Historically, the EMEA region is often seen as a “do more with less” area, where they don’t often spend big money on top-tier software licenses, instead working to fill gaps in open source software with processes and tools. And, in fact, respondents in EMEA were most likely (48% vs. 39% globally) to choose to “build on open source and run it myself” when it comes to provisioning tools. On the other hand, respondents in the Asia-Pacific region were most likely to have no plans to use provisioning tools. Additional regulatory concerns could be one factor behind this regional disparity.

The COVID-19 pandemic also impacted adoption of provisioning tools. Of the 54% of organizations that reported the pandemic accelerated their digital transformation initiatives with regards to shifts to cloud or multi-cloud adoption, infrastructure as code was the most likely area to be affected.

How

»Skills Shortage Is Top Provisioning Challenge

Even as adoption of multi-cloud architectures is expected to grow from 76% today to 86% in two years, key concerns remain. The most significant challenge hindering organizations’ ability to operationalize multi-cloud was a shortage of key skills, cited by 57% of respondents. Notably, the next most common response, inconsistent workflows across cloud environments, was cited by only 33% of respondents, a full 24 points behind skills shortages.

What

The survey also looked at cloud challenges by component, and taking a provisioning-centric view, staff and skilling issues were still the top concern, but the differences were much narrower.

What

Breaking the data down by industry shows the skills shortage is disproportionately impacting the public sector (25%) and retail/consumer goods industry (23%) the most.

»Open Source Provisioning Tools Are Most Popular

The survey showed that many organizations invested significant resources in their multi-cloud initiatives, including a third of respondents who budgeted $2 million or more per year (15% boasted $10 million+ cloud budgets!). Surprisingly, however, this did not always directly translate into greater spending on provisioning tools. Two-thirds of respondents use or plan to use some form of open source automation tooling for provisioning. Looked at another way, two-thirds of respondents are running their own provisioning tools instead of consuming a managed service.

Infrastructure

The relatively sparse interest in managed service offerings becomes even more interesting in the context of the top cloud challenges mentioned above. Adoption of Software-as-a-Service (SaaS) solutions would seem to be a natural response to skills shortages, complex manual processes, and a fast-moving environment. With a SaaS approach, teams could refocus their time away from maintaining and operating their custom tooling into automating the manual processes that appear to be slowing them down.

»Learn More

A deeper look at the role of open source software and managed services can be found in our blog post on When Free Isn’t Good Enough — Why Companies Buy Infrastructure Tools, and you can read all our survey analysis here. Or go straight to the source and check out the full HashiCorp State of Cloud Strategy Survey.


Source: HashiCorp Blog

Managing Credentials in Terraform Cloud & Enterprise

Managing Credentials in Terraform Cloud & Enterprise

There are several ways to manage credentials or other secret types in Terraform Cloud and Terraform Enterprise, either natively, or with purpose-built secrets management utilities like HashiCorp Vault, so this is a somewhat opinionated article that lists what I believe are currently the best options. Please note this is not a replacement for some other best practices, such as keeping your Terraform CLI and code up to date.

The described patterns follow some common principles. Credentials should be:

  • As unique as possible per workspace
  • Easy to rotate
  • As dynamic as possible
  • Protected with RBAC

There are also some nuances between Terraform Cloud and Terraform Enterprise that I will call out in each section.

In terms of security, both Cloud and Enterprise products encrypt their sensitive variables using the Vault Transit secrets engine and do not allow any external API call to decrypt these values. For more details please see the Data Security page.

If you still have questions by the end of this blog post, try attending one of our community office hours or booking a 1:1 technical session.

»Managing Credentials Using Only Terraform Workspaces

Using only Terraform workspaces, your cloud vendor’s Terraform provider, and the Terraform Cloud/Enterprise provider, you can set up a “Credentials” workspace that is able to generate new credentials and rotate the ones used by other workspaces. To avoid the secret zero problem, when setting up the credentials workspaces, you can make use of the Terraform Agents pattern described in the next section.

Note: In Terraform Cloud all workspaces live in the same Organization.

Terraform

Pros

  • There’s a separation between development environments.
  • It’s possible to use different types of secrets.

Cons

  • Requires a 3rd party API trigger to force a re-deployment of all “Credentials” workspaces.
  • Complex RBAC setup that does not exist in the free tier.
  • Secrets are stored in the “Credentials” workspace state.
  • “Credentials” workspaces require a Terraform Cloud/Enterprise User or Team token to access other workspaces that reside in another Organization.

»Managing Credentials Using Only Terraform Agents

Both Terraform Cloud Business tier and Terraform Enterprise support running your code using external agents. This feature is called Terraform Agents. Any cloud provider declared in your Terraform code is able to take advantage of the credentials set in the Terraform Agent environment, which means the credentials do not need to be set at the workspace level.

HashiCorp Solutions Engineer Andy Assareh has a repo and a recording to help walk you through this pattern.

Terraform

Pros

  • There’s a separation between development environments.
  • No credentials are set in project workspaces.
  • No credentials are stored in the workspace state.
  • One of the least complex patterns.

Cons

  • Only addresses cloud access credentials, not other kinds of secrets.
  • Requires Terraform Cloud Business tier.
  • Careful RBAC is required to ensure developers are not able to change Agent Pools in the workspace.
  • Difficult to scale and audit when using multiple accounts.
  • In Terraform Cloud Business tier, there are some limits on the number of Agents and Agent Pools an Organization can have.
  • Difficult to have a 1:1 mapping between workspace and Agent.

»Managing Credentials Using Vault

It’s no secret HashiCorp Vault is able to generate dynamic credentials for all major cloud vendors, databases, etc. For those who know about Vault, this integration with Terraform is the first solution they ask about, since secrets management is Vault’s primary use case. Here are a few patterns to make that integration work.

»Direct Integration with a Vault Plugin

There isn’t much to explain in this workflow. Most of the complexity is in setting up Vault authentication and some Terraform template code.

Terraform

Pros

Cons

  • Need to install a 3rd party plugin. More details are in the plugin repo.
  • Because we are using the Vault provider, secrets will be present in the state file.
  • Requires Terraform code changes to use the Vault provider.
  • Increase in complexity to set up Vault correctly.

»Integration with CI/CD

At the heart of this integration pattern, is the ability to confidently authenticate to Vault with an identity that is unique and combines workflow, repository, and branch/environment.

This is not the case with all CI/CD implementations, so I’m only mentioning two implementations where I know it’s possible.

Vault tokens should be very short-lived and linked to an entity, to restrict access in case any token gets leaked and allow for better auditing. Avoid storing Vault tokens as secrets!

Regarding the workflow, the CI/CD runner will need to authenticate to Vault and retrieve:

Pros

  • It’s possible to have 0 hardcoded or long-lived credentials, using only identities.
  • There’s a separation between development environments.
  • Different credentials for different branches.
  • Terraform Enterprise and Terraform Cloud credentials are not stored in Terraform state or the CI/CD platform.

Cons

  • Requires a CI/CD system able to assign an identity that combines a unique run id, workflow/pipeline, repository, and commit id.
  • May lead to secret sprawl.
  • Increase in pipeline complexity.

These pros and cons apply to the two subsections below.

»GitLab CI/CD Integration

GitLab runners are able to authenticate to Vault using the JWT auth backend, where you can configure separate roles for staging, dev, or production.

Once authentication has been solved, GitLab CI/CD will be able to retrieve the necessary secrets and interact with the Terraform Cloud/Terraform Enterprise API. Here’s an example.

For private repositories, you’ll need to have GitLab Premium.

Terraform,

»GitHub Actions Integration

For GitHub Actions to authenticate to Vault there are two options:

If you use AppRole, please make sure you also have a cron job set up to rotate the environment secrets, and use the “Repository Environments” to distinguish between production and non-production.

If you use the custom Vault GitHub Action authentication backend, right now it’s not possible to distinguish which branch is being executed.

»Pick the Solution that’s Right For You

There is no right answer or one-size-fits-all solution to managing your credentials and secrets within Terraform Cloud and Terraform Enterprise. This is highly dependent on your current requirements, environment, tools, etc, so it’s up to you to select the pattern that is best for you and your team, taking into account the pros and cons of each solution.

Please bear in mind that all of these solutions require you to plan and execute an efficient team/application onboarding process to be successful. To learn more about onboarding applications into Vault check out our other blog post Onboarding Applications to Vault Using Terraform: A Practical Guide or book a 1:1 session with one of our experts.


Source: HashiCorp Blog