Helm-manager

Project Layout

# List Generator
.
├── internal
│   ├── Chart.yaml
│   ├── templates
│   │   └── infra charts 的 ApplicationSets
│   └── values.yaml
├── stable
│   ├── Chart.yaml
│   ├── templates
│   │   └── infra charts 的 ApplicationSets
│   └── values.yaml
├── shared-values
│   └── infra charts 的共用 values
└── charts/
    └── prometheus/
    │   ├── templates/
    │   ├── Charts.yaml (with dependency chart)
    │   └── values.yaml
		└── infra charts/

• Generator 參數使用法：{{"{{"}} var_name {{"}}"}}
  • 第一層 render 是 helm 填入 values.yaml 的值
  • 第二層 render 是 ApplicationSet 填入 generators 的
• infra charts
  • ingress-nginx
    • client_a（裡面是這個 app 應該要有的 custom values）
    • mo
  • load-balancer-controller
    • xxx
  • karpenter
  • external-secrets
  • keda
  • image-updater
  • prometheus-stack

已知問題

• 會用到 generator 帶的變數的欄位要寫在 valuesObject 裡面（generator 參數傳不進 value files）
• list generator 不一定是最佳解
• 用 stable/internal branch 的優缺點
  • 之後要加服務時，要同時給兩個分支加 → 重工？
    • git basic，類似 release 到 staging/prod 的分支策略
• AppSet 長出來的 App 不能放在跟 AppSet 不同的 ns 裡
  • Solution:
    • AppSet 放在自己的 appset ns
    • ⚠️ 去訂閱那個 issue

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ingress-nginx
  namespace: {{ .Values.appset_ns }}
spec:
  generators:
  - list:
      elements:
      - namespace: client_a-internal
        name: client_a-internal
      - namespace: client_b-internal
        name: client_b-internal
  template:
    metadata:
      # https://github.com/argoproj/applicationset/issues/553#issuecomment-1075581614
      name: '{{"{{"}}namespace{{"}}"}}-ingress-nginx'
      namespace: '{{"{{"}}namespace{{"}}"}}'
    spec:
      project: {{ .Values.project }}
      source:
        repoURL: https://kubernetes.github.io/ingress-nginx
        chart: ingress-nginx
        targetRevision: 4.8.2
      destination:
        name: '{{"{{"}}name{{"}}"}}'
        namespace: kube-system

  # This sync policy pertains to the ApplicationSet, not to the Applications it creates.
  syncPolicy:
    # Determines whether the controller will delete Applications when an ApplicationSet is deleted.
    preserveResourcesOnDeletion: true

Generators

List Generator

• 基於 list 裡面的 element，為每個 element 分別長出各自的 App

• element 沒有必填欄位

• List Generator Example

  spec:
    generators:
    - list:
        elements:
          # v0.1.0 form - requires cluster/url keys:
          - cluster: engineering-dev
            url: https://kubernetes.default.svc
            values:
              additional: value
          # v0.2.0+ form - does not require cluster/URL keys
          # (but they are still supported).
          - staging: "true"
            gitRepo: https://kubernetes.default.svc   
  # (...)

• 可以用 matrix_generator 混合 list + git generator

  • 用 git generator 指到包含 list element 的檔案
  • 用 list generator 的 elementsYaml 解析上述檔案的 yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: elementsYaml
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - matrix:
      generators:
      - git:
          repoURL: https://github.com/argoproj/argo-cd.git
          revision: HEAD
          files:
          - path: applicationset/examples/list-generator/list-elementsYaml-example.yaml
      - list:
          elementsYaml: "{{ .key.components | toJson }}"
  template:
    metadata:
      name: '{{.name}}'
    spec:
      project: default
      sources:
        - chart: '{{.chart}}'
          repoURL: '{{.repoUrl}}'
          targetRevision: '{{.version}}'
          helm:
            releaseName: '{{.releaseName}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.namespace}}'
---
# list-elementsYaml-example.yaml
key:
  components:
    - name: component1
      chart: podinfo
      version: "6.3.2"
      releaseName: component1
      repoUrl: "https://stefanprodan.github.io/podinfo"
      namespace: component1
    - name: component2
      chart: podinfo
      version: "6.3.3"
      releaseName: component2
      repoUrl: "ghcr.io/stefanprodan/charts"
      namespace: component2

Cluster Generator

• 根據 argo 已經管理的 clusters 作為 generators

  • 被 argo 管理的 clusters 儲存在 argo ns 的 secret 裡，AppSet 拿 secret 當作分辨 available cluster 的依據 → argo-cluster-secret-${project_name}
  • 為避免與 external-secret 的 ClusterSecret 搞混，以下會稱此 secret 為 argo cluster secret

• 可用的 parameters

  • name / nameNormalized
  • server
  • metadata.labels.<key> (for each label in the Secret)
  • metadata.annotations.<key> (for each annotation in the Secret)

• 可以用 values 傳入自定義的 parameters

• Cluster GeneratorExample

  spec:
    goTemplate: true
    goTemplateOptions: ["missingkey=error"]
    generators:
    - clusters: {} # Automatically use all clusters defined within Argo CD

• 可以用 Label Selector 來選取特定的 cluster，會對應到含有特定 label 的 argo cluster secret

  • 要包含 local cluster 的話，可以幫 local cluster 建立 argo cluster secret

Git Generator

分為 directory 與 file 兩種 generator

Git Directory Generator

• 可以長出指定路徑下各個 folder 作為 Application

• 可以使用 exclude 指定特定 folder 不要被生成 Application [details]

• 要讀取 repo root，path 可以放 ’*’

• 可以用 values 傳入自定義的 parameters，呼叫時需使用 {{.values.xxx}}

• 使用 git directory generator 時可以使用的 parameters

  • {{.path.path}}: Git repo 中符合 directory path wildcard 規則的 path
  • {{index .path.segments n}}: 所有符合 path wildcard 的 path 陣列 (n - array index)
  • {{.path.basename}}: path 中最右邊的一段 (e.g. /directory/directory2 → directory2).
  • {{.path.basenameNormalized}}: 把 path.basename 中不支援的字元替換成 - (e.g. path.basename = directory_2 → directory-2)

• Generator Example

  generators:
    - git:
        repoURL: https://github.com/argoproj/argo-cd.git
        revision: HEAD
        directories:
        - path: applicationset/examples/git-generator-directory/cluster-addons/*
        - path: applicationset/examples/git-generator-directory/excludes/cluster-addons/exclude-helm-guestbook
          exclude: true
        values:
          cluster: '{{.branch}}-{{.path.basename}}'

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - git:
      repoURL: https://github.com/argoproj/argo-cd.git
      revision: HEAD
      directories:
      - path: applicationset/examples/git-generator-directory/cluster-addons/*
  template:
    metadata:
      name: '{{.path.basename}}'
    spec:
      project: "my-project"
      source:
        repoURL: https://github.com/argoproj/argo-cd.git
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.path.basename}}'
      syncPolicy:
        syncOptions:
        - CreateNamespace=true
---
# Directory tree under cluster-addons/ (path)
├── argo-workflows
│   ├── kustomization.yaml
│   └── namespace-install.yaml
└── prometheus-operator
    ├── Chart.yaml
    ├── README.md
    ├── requirements.yaml
    └── values.yaml

Git File Generator

• 為每個符合規則的檔案各自生成 Application

• 可以用 values 傳入自定義的 parameters

• 可以傳入 json/yaml 兩種檔案，被 generator 讀取之後會變成 template parameters

  → 所以 goTemplate 要設定為 true

• 可用的 parameters 與 directory generator 類似

• Example

  spec:
    goTemplate: true
    goTemplateOptions: ["missingkey=error"]
    generators:
    - git:
        repoURL: https://github.com/argoproj/argo-cd.git
        revision: HEAD
        files:
        - path: "applicationset/examples/git-generator-files-discovery/cluster-config/**/config.json"
  
  # Directory tree under cluster-addons/ (path)
  ├── cluster-config
  │   └── engineering
  │       ├── dev
  │       │   └── config.json
  │       └── prod
  │           └── config.json
  └── git-generator-files.yaml
  
  # Content inside config.json
  {
    "aws_account": "123456",
    "asset_id": "11223344",
    "cluster": {
      "owner": "cluster-admin@company.com",
      "name": "engineering-dev",
      "address": "https://1.2.3.4"
    }
  }

Webhook Config

預設 AppSet 會每三分鐘 poll 一次 git repo，如果想要避免這個 delay 的話，可以透過設定 webhook 讓 gitlab/github 主動戳 argo

SCM Provider Generator ¶

• 利用 SCMaaS provider API (eg GitHub) 自動偵測 organization 有沒有新的 repo
• 適用於把 microservices 切到各個 repo 的架構

Pull Request Generator

• 需要搭配 SCMaaS provider API 使用

• 跟 git generator 一樣，PR generator 可以設定 webhook 避免三分鐘才 poll 一次的 delay

• Gitlab PR Generator 使用法

  apiVersion: argoproj.io/v1alpha1
  kind: ApplicationSet
  metadata:
    name: myapps
  spec:
    goTemplate: true
    goTemplateOptions: ["missingkey=error"]
    generators:
    - pullRequest:
        gitlab:
          # The GitLab project.
          project: myproject
          # For self-hosted GitLab (optional)
          api: https://git.example.com/
          # Reference to a Secret containing an access token. (optional)
          tokenRef:
            secretName: gitlab-token
            key: token
          # Labels is used to filter the MRs that you want to target. (optional)
          labels:
          - preview
          # MR state is used to filter MRs only with a certain state. (optional)
          pullRequestState: opened
          # If true, skips validating the SCM provider's TLS certificate - useful for self-signed certificates.
          insecure: false
        requeueAfterSeconds: 1800
    template:
    # ...

Post Selector All Generator

• 在最後生成的 parameters 裡面選擇只保留符合的參數組

• Example

  generators:
  - list:
      elements:
        - cluster: engineering-dev
          url: https://kubernetes.default.svc
          env: staging
        - cluster: engineering-prod
          url: https://kubernetes.default.svc
          env: prod
    selector: # -> Post selector
      matchLabels:
        env: staging
  
  # 實際上 valid 的 parameters
  - cluster: engineering-dev
    url: https://kubernetes.default.svc
    env: staging

Matrix Generator

• 可以將兩個 generator 生成的 parameters 做結合，並對兩個 generator 的所有組合作迭代

  • Example: Git Directory + Cluster

    • Git Directory → Premetheus Operator(PO) & Grafana Dashboard(GD)
    • Cluster → client_a-internal(d1) & client_b-internal(m1)

    $$ → (PO, GD) \times(d1, m1)=(PO, d1), (GD, d1), (PO, m1), (GD, m1) $$

• Example Combination
  • SCM + Cluster：讀取某個 org 把新的 repo 佈在所有 cluster 上
  • Git File + List：透過 config files 部署 apps 到特定 list 中的所有 cluster 裡
  • Git Directory + Cluster Decision Resource：把在 git dir 中的 app 部署到透過 external custom resource 提供的 cluster 中
• Git Directory: infra charts using dependency chart with custom values
• Cluster: list out all remote clusters

Merge Generator

• 可以將多個 generator 生成的 parameter 結合，但會依據 merge keys 做合併

• 後面的 generator 會覆蓋掉前面的設定

  generators:
    # merge 'parent' generator
    - merge:
        mergeKeys:
          - server
        generators:
          - clusters:
              values:
                kafka: 'true'
                redis: 'false'
          # For clusters with a specific label, enable Kafka.
          - clusters:
              selector:
                matchLabels:
                  use-kafka: 'false'
              values:
                kafka: 'false'
          # For a specific cluster, enable Redis.
          - list:
              elements: 
                - server: https://2.4.6.8
                  values.redis: 'true'

1. 拿到所有 clusters, with value kafka: true & redis: false

   → 生成以下 parameter 組合：

   - name: staging
   	server: https://1.2.3.4
   	values.kafka: 'true'
   	values.redis: 'false'
   - name: production
   	server: https://2.4.6.8
   	values.kafka: 'true'
   	values.redis: 'false'

2. 選定 argo cluster secret 的 label 符合 use-kafka: false 的那組 parameters,

   with value kafka: false

   → 生成以下 parameter 組合：

   # Staging cluster has the label `use-kafka: false`
   - name: staging
     server: https://1.2.3.4
     values.kafka: 'false'
     values.redis: 'false'
   
   - name: production
     server: https://2.4.6.8
     values.kafka: 'true'
     values.redis: 'false'

3. 選定 server: https://2.4.6.8 的 parameter，代入 value redis: true

   → 生成以下 parameter 組合：

   - name: staging
   	server: https://1.2.3.4
   	values.kafka: 'false'
   	values.redis: 'false'
   - name: production
   	server: https://2.4.6.8
   	values.kafka: 'true'
   	values.redis: 'true'

   generators:
     # merge 'parent' generator:
     # Use the selector set by both child generators to combine them.
     - merge:
         mergeKeys:
           # Note that this would not work with goTemplate enabled,
           # nested merge keys are not supported there.
           - values.selector
         generators:
           # Assuming, all configured clusters have a label for their location:
           # Set the selector to this location.
           - clusters:
               values:
                 selector: '{{index .metadata.labels "location"}}'
           # The git repo may have different directories which correspond to the
           # cluster locations, using these as a selector.
           - git:
               repoURL: https://github.com/argoproj/argocd-example-apps/
               revision: HEAD
               directories:
               - path: '*'
               values:
                 selector: '{{.path.path}}'

• 假設 cluster 叫做 germany01，location label = Germany，且 git repo 有個 directory 叫 Hamburger，此時生成的 parameter 如下：

  # From the cluster generator
  - name: germany01
    server: https://1.2.3.4
    # From the git generator
    path: Hamburger
    # Combining selector with the merge generator
    values.selector: 'Hamburger' # original is 'Germany'
    # More values from cluster & git generator
    # […]

• merge in merge/matrix + post selector 要搭配 enable spec.applyNestedSelectors 才能用

🚧 Cluster Decision Resource Generator

• 針對自己寫的 CRD 的 generator

🚧 Plugin Generator

• 自定義的 Generator