<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://arshadmehmood.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://arshadmehmood.com/" rel="alternate" type="text/html" /><updated>2026-04-01T17:09:54+00:00</updated><id>https://arshadmehmood.com/feed.xml</id><title type="html">Arshad Mehmood</title><subtitle>Personal blog by Arshad Mehmood. Software Engineer specializing in web and mobile development. Sharing insights on Android, DevOps, and modern software engineering practices.</subtitle><author><name>Arshad Mehmood</name></author><entry><title type="html">Complete Guide to SSH: From Basics to Advanced</title><link href="https://arshadmehmood.com/devops/complete-guide-to-ssh-from-basics-to-advanced/" rel="alternate" type="text/html" title="Complete Guide to SSH: From Basics to Advanced" /><published>2026-04-01T00:00:00+00:00</published><updated>2026-04-01T00:00:00+00:00</updated><id>https://arshadmehmood.com/devops/complete-guide-to-ssh-from-basics-to-advanced</id><content type="html" xml:base="https://arshadmehmood.com/devops/complete-guide-to-ssh-from-basics-to-advanced/"><![CDATA[<p>SSH (Secure Shell) is the foundation of secure remote access to servers and systems. This guide covers everything from basics to advanced usage, with practical examples you can use immediately.</p>

<h2 id="table-of-contents">Table of Contents</h2>

<ul>
  <li><a href="#ssh-basics">SSH Basics</a></li>
  <li><a href="#authentication-methods">Authentication Methods</a></li>
  <li><a href="#key-management">Key Management</a></li>
  <li><a href="#remote-command-execution">Remote Command Execution</a></li>
  <li><a href="#file-transfers">File Transfers</a></li>
  <li><a href="#advanced-techniques">Advanced Techniques</a></li>
  <li><a href="#troubleshooting">Troubleshooting</a></li>
  <li><a href="#security-best-practices">Security Best Practices</a></li>
</ul>

<h2 id="ssh-basics">SSH Basics</h2>

<p>SSH lets you securely connect to remote servers, execute commands, and transfer files over encrypted connections.</p>

<h3 id="basic-connection-syntax">Basic Connection Syntax</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="o">[</span>options] <span class="o">[</span>user@]hostname
ssh user@192.168.1.100
ssh root@example.com
</code></pre></div></div>

<p>The default SSH port is <strong>22</strong>, but servers often use custom ports for security.</p>

<p><strong>Full TIL:</strong> <a href="/today-i-learned/ssh/ssh-connection-basics/">SSH Connection Basics: Password, Keys, and Parameters</a></p>

<h2 id="authentication-methods">Authentication Methods</h2>

<p>SSH supports two primary authentication methods:</p>

<h3 id="1-password-authentication">1. Password Authentication</h3>

<p>The simplest method—SSH prompts for password each time:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh user@hostname
<span class="c"># Prompted for password</span>
</code></pre></div></div>

<p><strong>Drawbacks:</strong></p>
<ul>
  <li>Requires typing password repeatedly</li>
  <li>Vulnerable to brute-force attacks</li>
  <li>Poor for automation and scripts</li>
</ul>

<h3 id="2-key-based-authentication-recommended">2. Key-Based Authentication (Recommended)</h3>

<p>Uses cryptographic key pairs instead of passwords. Much more secure and convenient.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-i</span> ~/.ssh/id_ed25519 user@hostname
</code></pre></div></div>

<p><strong>Benefits:</strong></p>
<ul>
  <li>✅ No password to remember</li>
  <li>✅ Can’t be brute-forced</li>
  <li>✅ Perfect for scripts and automation</li>
  <li>✅ Passphrase-protected keys for extra security</li>
</ul>

<p><strong>Full TIL:</strong> <a href="/today-i-learned/ssh/ssh-connection-basics/">SSH Connection Basics</a></p>

<h2 id="key-management">Key Management</h2>

<h3 id="generating-ssh-keys">Generating SSH Keys</h3>

<p>Modern best practice: use Ed25519 keys (faster, smaller, more secure than RSA):</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-keygen <span class="nt">-t</span> ed25519 <span class="nt">-C</span> <span class="s2">"your-email@example.com"</span>
<span class="c"># Or traditional RSA:</span>
ssh-keygen <span class="nt">-t</span> rsa <span class="nt">-b</span> 4096
</code></pre></div></div>

<p>Creates two files:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">~/.ssh/id_ed25519</code> — <strong>Private key</strong> (keep secret, <code class="language-plaintext highlighter-rouge">chmod 600</code>)</li>
  <li><code class="language-plaintext highlighter-rouge">~/.ssh/id_ed25519.pub</code> — <strong>Public key</strong> (share with servers)</li>
</ul>

<h3 id="using-specific-keys">Using Specific Keys</h3>

<p>By default SSH uses keys from <code class="language-plaintext highlighter-rouge">~/.ssh/</code>. To use a specific key:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-i</span> /path/to/private/key user@hostname
ssh <span class="nt">-i</span> ~/.ssh/custom_key <span class="nt">-p</span> 2222 user@hostname
</code></pre></div></div>

<h3 id="ssh-agent-never-type-your-passphrase-again">SSH Agent: Never Type Your Passphrase Again</h3>

<p>SSH agent stores decrypted keys in memory, so you enter your passphrase once per session instead of per connection.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Start the agent (usually automatic)</span>
<span class="nb">eval</span> <span class="s2">"</span><span class="si">$(</span>ssh-agent <span class="nt">-s</span><span class="si">)</span><span class="s2">"</span>

<span class="c"># Add your key (prompts for passphrase once)</span>
ssh-add ~/.ssh/id_ed25519

<span class="c"># Now use SSH without typing passphrase</span>
ssh user@hostname  <span class="c"># No passphrase needed!</span>
</code></pre></div></div>

<p><strong>Useful agent commands:</strong></p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-add <span class="nt">-l</span>            <span class="c"># List loaded keys</span>
ssh-add <span class="nt">-d</span> ~/.ssh/id  <span class="c"># Remove specific key</span>
ssh-add <span class="nt">-D</span>            <span class="c"># Remove all keys</span>
</code></pre></div></div>

<p><strong>Full TIL:</strong> <a href="/today-i-learned/ssh/using-ssh-agent-to-manage-keys/">Using SSH Agent to Manage Keys</a></p>

<h2 id="remote-command-execution">Remote Command Execution</h2>

<p>Run commands on remote servers without opening an interactive shell:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh user@hostname <span class="s2">"command"</span>
ssh user@hostname <span class="s2">"ls -la /tmp"</span>
ssh user@hostname <span class="s2">"whoami"</span>
</code></pre></div></div>

<h3 id="multiple-commands">Multiple Commands</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Using &amp;&amp; (only runs next if previous succeeds)</span>
ssh user@hostname <span class="s2">"cd /tmp &amp;&amp; ls -la"</span>

<span class="c"># Using ; (runs regardless)</span>
ssh user@hostname <span class="s2">"uptime; df -h"</span>

<span class="c"># Multi-line script</span>
ssh user@hostname <span class="o">&lt;&lt;</span> <span class="sh">'</span><span class="no">EOF</span><span class="sh">'
cd /tmp
ls -la
whoami
</span><span class="no">EOF
</span></code></pre></div></div>

<h3 id="capturing-output">Capturing Output</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh user@hostname <span class="s2">"ls -la /tmp"</span> <span class="o">&gt;</span> local-file.txt
ssh user@hostname <span class="s2">"cat /var/log/syslog"</span> | <span class="nb">grep</span> <span class="s2">"error"</span> | <span class="nb">wc</span> <span class="nt">-l</span>
</code></pre></div></div>

<h3 id="useful-patterns">Useful Patterns</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Check if service is running</span>
ssh user@hostname <span class="s2">"systemctl is-active nginx"</span>

<span class="c"># Run with sudo (prompts for password)</span>
ssh user@hostname <span class="s2">"sudo systemctl restart nginx"</span>

<span class="c"># Execute local script on remote server</span>
ssh user@hostname &lt; local-script.sh

<span class="c"># Background task</span>
ssh user@hostname <span class="s2">"nohup long-task &gt; output.log 2&gt;&amp;1 &amp;"</span>
</code></pre></div></div>

<p><strong>Full TIL:</strong> <a href="/today-i-learned/ssh/executing-remote-commands-with-ssh/">Executing Remote Commands with SSH</a></p>

<h2 id="file-transfers">File Transfers</h2>

<h3 id="scp-simple-file-copy">SCP: Simple File Copy</h3>

<p>SCP is straightforward for one-off transfers:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Local to remote</span>
scp ./file.txt user@hostname:/path/to/remote/

<span class="c"># Remote to local</span>
scp user@hostname:/path/to/file.txt ./local-folder/

<span class="c"># With custom port and key</span>
scp <span class="nt">-i</span> ~/.ssh/key <span class="nt">-P</span> 2222 ./file.txt user@hostname:/path/
</code></pre></div></div>

<p><strong>Best for:</strong></p>
<ul>
  <li>Quick, small file transfers</li>
  <li>Simple one-time operations</li>
</ul>

<p><strong>Full TIL:</strong> <a href="/today-i-learned/ssh/using-scp-for-file-transfers/">Using SCP for File Transfers</a></p>

<h3 id="rsync-smart-synchronization">RSYNC: Smart Synchronization</h3>

<p>RSYNC transfers only changed parts and can resume interrupted transfers:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Local to remote</span>
rsync <span class="nt">-avz</span> ./folder/ user@hostname:/path/to/remote/

<span class="c"># Remote to local</span>
rsync <span class="nt">-avz</span> user@hostname:/path/to/folder/ ./local-folder/

<span class="c"># Single file</span>
rsync <span class="nt">-avz</span> ./file.txt user@hostname:/path/
</code></pre></div></div>

<p><strong>Common options:</strong></p>

<table>
  <thead>
    <tr>
      <th>Option</th>
      <th>Purpose</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">-a</code></td>
      <td>Archive mode (preserves permissions, times)</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">-v</code></td>
      <td>Verbose output</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">-z</code></td>
      <td>Compress during transfer</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">--delete</code></td>
      <td>Delete files in destination not in source</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">--dry-run</code></td>
      <td>Preview without copying</td>
    </tr>
  </tbody>
</table>

<p><strong>Best for:</strong></p>
<ul>
  <li>Large backups</li>
  <li>Repeated syncs</li>
  <li>Resumable transfers</li>
</ul>

<p><strong>Full TIL:</strong> <a href="/today-i-learned/ssh/using-rsync-for-file-transfers/">Using RSYNC for File Transfers</a></p>

<h3 id="scp-vs-rsync">SCP vs RSYNC</h3>

<table>
  <thead>
    <tr>
      <th>Feature</th>
      <th>SCP</th>
      <th>RSYNC</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Speed (first transfer)</td>
      <td>Same</td>
      <td>Same</td>
    </tr>
    <tr>
      <td>Incremental</td>
      <td>❌ Full copy</td>
      <td>✅ Only changes</td>
    </tr>
    <tr>
      <td>Resume</td>
      <td>❌ Restarts</td>
      <td>✅ Continues</td>
    </tr>
    <tr>
      <td>Complexity</td>
      <td>Simple</td>
      <td>More options</td>
    </tr>
    <tr>
      <td>Best use</td>
      <td>Quick transfers</td>
      <td>Backups, automation</td>
    </tr>
  </tbody>
</table>

<h2 id="advanced-techniques">Advanced Techniques</h2>

<h3 id="port-forwarding">Port Forwarding</h3>

<p><strong>Local port forwarding</strong> — Access remote service through local port:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-L</span> 8080:localhost:3000 user@hostname <span class="nt">-N</span> <span class="nt">-f</span>
<span class="c"># Now: localhost:8080 → remote:3000</span>
curl localhost:8080  <span class="c"># Accesses remote service</span>
</code></pre></div></div>

<p><strong>Remote port forwarding</strong> — Expose local service to remote network:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-R</span> 8080:localhost:3000 user@hostname <span class="nt">-N</span> <span class="nt">-f</span>
<span class="c"># Remote can access: localhost:8080 → your local:3000</span>
</code></pre></div></div>

<h3 id="custom-ssh-config">Custom SSH Config</h3>

<p>Create <code class="language-plaintext highlighter-rouge">~/.ssh/config</code> for easier, personalized connections:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host production
    Hostname example.com
    User deploy
    Port 2222
    IdentityFile ~/.ssh/production_key
    ServerAliveInterval 60
    LogLevel QUIET

Host github
    Hostname github.com
    User git
    IdentityFile ~/.ssh/github_key
</code></pre></div></div>

<p>Now use short names:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh production
ssh github  <span class="c"># For git operations</span>
</code></pre></div></div>

<h3 id="proxyjump-jump-through-bastion-host">ProxyJump: Jump Through Bastion Host</h3>

<p>Connect through a bastion/jump host:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-J</span> bastion@jump-host internal@internal-server
<span class="c"># Or in ~/.ssh/config</span>
Host internal-server
    Hostname 10.0.1.5
    ProxyJump jump-host
</code></pre></div></div>

<h3 id="quiet-mode">Quiet Mode</h3>

<p>Suppress banners and diagnostic messages:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-q</span> user@hostname
ssh <span class="nt">-q</span> user@hostname <span class="s2">"command"</span>
</code></pre></div></div>

<p><strong>Full TIL:</strong> <a href="/today-i-learned/ssh/ssh-connection-without-welcome-message-or-banner/">SSH Connection Without Welcome Message</a></p>

<h2 id="troubleshooting">Troubleshooting</h2>

<h3 id="host-key-changed-warning">Host Key Changed Warning</h3>

<p>If you see this warning:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
</code></pre></div></div>

<p>This happens after server rebuilds or IP changes. Remove the old key:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-keygen <span class="nt">-R</span> hostname.com
<span class="c"># Then reconnect</span>
ssh user@hostname.com
</code></pre></div></div>

<p><strong>Full TIL:</strong> <a href="/today-i-learned/ssh/fix-remote-host-identification-has-changed-warning-with-ssh-keygen-r/">Fix REMOTE HOST IDENTIFICATION HAS CHANGED</a></p>

<h3 id="debug-connection-issues">Debug Connection Issues</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-v</span> user@hostname          <span class="c"># Verbose output</span>
ssh <span class="nt">-vv</span> user@hostname         <span class="c"># More verbose</span>
ssh <span class="nt">-vvv</span> user@hostname        <span class="c"># Maximum debug info</span>
</code></pre></div></div>

<h3 id="common-issues">Common Issues</h3>

<p><strong>Permission Denied:</strong></p>
<ul>
  <li>Check key file permissions: <code class="language-plaintext highlighter-rouge">chmod 600 ~/.ssh/id_ed25519</code></li>
  <li>Verify server authorized_keys contains your public key</li>
</ul>

<p><strong>Connection Timeout:</strong></p>
<ul>
  <li>Check if port is correct: <code class="language-plaintext highlighter-rouge">ssh -p port ...</code></li>
  <li>Verify firewall allows SSH traffic</li>
  <li>Test connectivity: <code class="language-plaintext highlighter-rouge">nc -zv hostname port</code></li>
</ul>

<p><strong>Wrong Key Being Used:</strong></p>
<ul>
  <li>Explicitly specify: <code class="language-plaintext highlighter-rouge">ssh -i /path/to/key ...</code></li>
  <li>List available keys: <code class="language-plaintext highlighter-rouge">ssh-add -l</code></li>
</ul>

<h2 id="security-best-practices">Security Best Practices</h2>

<h3 id="1-use-key-based-authentication">1. Use Key-Based Authentication</h3>

<p>Never rely on passwords for SSH. Keys are exponentially more secure.</p>

<h3 id="2-protect-your-private-keys">2. Protect Your Private Keys</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Ensure proper permissions</span>
<span class="nb">chmod </span>600 ~/.ssh/id_<span class="k">*</span>

<span class="c"># Use passphrase-protected keys</span>
ssh-keygen <span class="nt">-p</span> <span class="nt">-f</span> ~/.ssh/id_ed25519  <span class="c"># Change passphrase</span>
</code></pre></div></div>

<h3 id="3-use-ed25519-keys-not-rsa">3. Use Ed25519 Keys (Not RSA)</h3>

<ul>
  <li>✅ Ed25519: Faster, smaller, newer, more secure</li>
  <li>⚠️ RSA: Still secure, but older (use 4096-bit minimum)</li>
</ul>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-keygen <span class="nt">-t</span> ed25519 <span class="nt">-C</span> <span class="s2">"your-email@example.com"</span>
</code></pre></div></div>

<h3 id="4-limit-ssh-access-on-servers">4. Limit SSH Access on Servers</h3>

<p>Edit <code class="language-plaintext highlighter-rouge">/etc/ssh/sshd_config</code> on your servers:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Disable password auth</span>
PasswordAuthentication no

<span class="c"># Disable root login</span>
PermitRootLogin no

<span class="c"># Change default port (security through obscurity is weak, but helps)</span>
Port 2222

<span class="c"># Limit login attempts</span>
MaxAuthTries 3
MaxSessions 10
</code></pre></div></div>

<p>Restart SSH: <code class="language-plaintext highlighter-rouge">sudo systemctl restart ssh</code> or <code class="language-plaintext highlighter-rouge">sudo systemctl restart sshd</code></p>

<h3 id="5-use-ssh-agent-instead-of-unencrypted-keys">5. Use SSH Agent Instead of Unencrypted Keys</h3>

<p>Never use SSH keys without passphrases. Let SSH agent handle the passphrase.</p>

<h3 id="6-monitor-ssh-activity">6. Monitor SSH Activity</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># View recent SSH attempts</span>
<span class="nb">grep</span> <span class="s2">"sshd"</span> /var/log/auth.log | <span class="nb">tail</span> <span class="nt">-20</span>

<span class="c"># See who's currently connected</span>
<span class="nb">who
</span>w

<span class="c"># Check SSH public keys authorized on this system</span>
<span class="nb">cat</span> ~/.ssh/authorized_keys
</code></pre></div></div>

<h3 id="7-rotate-ssh-keys-periodically">7. Rotate SSH Keys Periodically</h3>

<p>Even with secure keys, rotate them every 1-2 years as a best practice.</p>

<h2 id="summary">Summary</h2>

<p>SSH is powerful, but only when used correctly. Key takeaways:</p>

<ol>
  <li><strong>Always use key-based auth</strong> never trust passwords for SSH</li>
  <li><strong>Use Ed25519 keys</strong> they’re faster, smaller, and more secure</li>
  <li><strong>Protect your private keys</strong> with passphrases</li>
  <li><strong>Use SSH agent</strong> for convenience without sacrificing security</li>
  <li><strong>Keep your <code class="language-plaintext highlighter-rouge">~/.ssh</code> directory private</strong> (<code class="language-plaintext highlighter-rouge">chmod 700 ~/.ssh</code>)</li>
  <li><strong>Monitor SSH access</strong> on your servers regularly</li>
</ol>

<h2 id="learning-resources">Learning Resources</h2>

<p>All the TILs referenced in this post (concise, focused guides):</p>

<ul>
  <li><a href="/today-i-learned/ssh/ssh-connection-basics/">SSH Connection Basics</a> — password, keys, parameters</li>
  <li><a href="/today-i-learned/ssh/using-ssh-agent-to-manage-keys/">SSH Agent</a> — key management</li>
  <li><a href="/today-i-learned/ssh/executing-remote-commands-with-ssh/">Remote Command Execution</a> — running commands</li>
  <li><a href="/today-i-learned/ssh/using-scp-for-file-transfers/">SCP Transfers</a> — simple file copy</li>
  <li><a href="/today-i-learned/ssh/using-rsync-for-file-transfers/">RSYNC Transfers</a> — smart synchronization</li>
  <li><a href="/today-i-learned/ssh/fix-remote-host-identification-has-changed-warning-with-ssh-keygen-r/">Fixing Host Key Warnings</a></li>
  <li><a href="/today-i-learned/ssh/ssh-connection-without-welcome-message-or-banner/">Quiet Mode</a></li>
</ul>

<h2 id="future-topics">Future Topics</h2>

<p>Topics I plan to learn and add later:</p>

<ul>
  <li>Host key verification and strict host checking</li>
  <li>Copying public keys with <code class="language-plaintext highlighter-rouge">ssh-copy-id</code></li>
  <li>Advanced SSH config patterns (<code class="language-plaintext highlighter-rouge">IdentitiesOnly</code>, <code class="language-plaintext highlighter-rouge">Include</code>, multiple keys)</li>
  <li>Connection multiplexing (<code class="language-plaintext highlighter-rouge">ControlMaster</code>, <code class="language-plaintext highlighter-rouge">ControlPath</code>, <code class="language-plaintext highlighter-rouge">ControlPersist</code>)</li>
  <li>Agent forwarding and security implications</li>
  <li>Keepalive tuning (<code class="language-plaintext highlighter-rouge">ServerAliveInterval</code>, <code class="language-plaintext highlighter-rouge">ServerAliveCountMax</code>)</li>
  <li>Automation-focused flags (<code class="language-plaintext highlighter-rouge">BatchMode</code>, robust timeout handling)</li>
  <li>Server hardening extras (<code class="language-plaintext highlighter-rouge">AllowUsers</code>, <code class="language-plaintext highlighter-rouge">Fail2ban</code>, firewall rules)</li>
  <li>Hardware-backed SSH keys (<code class="language-plaintext highlighter-rouge">ed25519-sk</code>, <code class="language-plaintext highlighter-rouge">ecdsa-sk</code>)</li>
</ul>

<h2 id="next-steps">Next Steps</h2>

<ul>
  <li><strong>Review your SSH setup:</strong> Check your keys and permissions</li>
  <li><strong>Configure SSH config:</strong> Create <code class="language-plaintext highlighter-rouge">~/.ssh/config</code> for frequently used hosts</li>
  <li><strong>Audit server access:</strong> Review who has SSH keys on your servers</li>
  <li><strong>Run SSH agent on startup:</strong> Add to your shell profile for seamless key management</li>
</ul>

<hr />]]></content><author><name>Arshad Mehmood</name></author><category term="DevOps" /><category term="ssh" /><category term="security" /><category term="linux" /><category term="terminal" /><category term="remote-access" /><summary type="html"><![CDATA[Master SSH from connection basics to advanced file transfers. Learn authentication methods, key management, remote command execution, and security best practices.]]></summary></entry><entry><title type="html">How Fail2ban Locked Me Out of My Own Server (Traefik + Authentik)</title><link href="https://arshadmehmood.com/devops/how-fail2ban-locked-me-out-of-my-own-server-traefik-authentik/" rel="alternate" type="text/html" title="How Fail2ban Locked Me Out of My Own Server (Traefik + Authentik)" /><published>2026-04-01T00:00:00+00:00</published><updated>2026-04-01T00:00:00+00:00</updated><id>https://arshadmehmood.com/devops/how-fail2ban-locked-me-out-of-my-own-server-traefik-authentik</id><content type="html" xml:base="https://arshadmehmood.com/devops/how-fail2ban-locked-me-out-of-my-own-server-traefik-authentik/"><![CDATA[<p>I use Fail2ban to protect my server, but after wiring things around Traefik + Authentik and tightening rules too aggressively, I ended up banning my own public IP and lost SSH access.</p>

<p>This post documents what happened, how I recovered, and what I changed so it does not happen again.</p>

<h2 id="what-happened">What happened</h2>

<ul>
  <li>I enabled Fail2ban protection not only for SSH, but also for Traefik access logs.</li>
  <li>During testing, I hit multiple blocked/scanner-like URLs and authentication flows while Traefik + Authentik were in front.</li>
  <li>My custom Traefik jail treated those requests as malicious behavior and banned my current public IP.</li>
  <li>Because that was my admin IP at the time, I effectively locked myself out.</li>
  <li>SSH stopped working from my machine, so I could not log in normally.</li>
</ul>

<p>At that point, I only had one way in: provider-side remote console shell.</p>

<h2 id="symptoms-i-saw">Symptoms I saw</h2>

<ul>
  <li>SSH connection attempts timed out or failed immediately.</li>
  <li>Repeated prompts with no successful login.</li>
  <li>Fail2ban service was running, but I was effectively blocked.</li>
</ul>

<p>Commands that helped verify from console:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>fail2ban-client status
<span class="nb">sudo </span>fail2ban-client status sshd
<span class="c"># Check your custom proxy/auth jails by name</span>
<span class="nb">sudo </span>fail2ban-client status &lt;proxy_or_auth_jail&gt;
<span class="nb">sudo </span>iptables <span class="nt">-S</span> | <span class="nb">grep </span>f2b
</code></pre></div></div>

<p>If your distro uses nftables, inspect rules there instead:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>nft list ruleset | <span class="nb">grep</span> <span class="nt">-i</span> f2b
</code></pre></div></div>

<h2 id="how-i-recovered-access">How I recovered access</h2>

<p>I logged in via my cloud provider’s emergency/serial console, then:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Check active jails and banned IPs</span>
<span class="nb">sudo </span>fail2ban-client status
<span class="nb">sudo </span>fail2ban-client status sshd
<span class="nb">sudo </span>fail2ban-client status &lt;proxy_or_auth_jail&gt;

<span class="c"># Fastest recovery: unban from all jails at once</span>
<span class="nb">sudo </span>fail2ban-client unban &lt;MY_PUBLIC_IP&gt;

<span class="c"># If needed, unban per jail</span>
<span class="nb">sudo </span>fail2ban-client <span class="nb">set </span>sshd unbanip &lt;MY_PUBLIC_IP&gt;
<span class="nb">sudo </span>fail2ban-client <span class="nb">set</span> &lt;proxy_or_auth_jail&gt; unbanip &lt;MY_PUBLIC_IP&gt;
</code></pre></div></div>

<p>After unbanning, SSH access worked again from my machine.</p>

<h2 id="the-configuration-mistakes">The configuration mistakes</h2>

<p>The root issue was not Fail2ban itself. It was my configuration:</p>

<ul>
  <li>Aggressive or test-phase ban settings while still validating proxy/auth behavior.</li>
  <li>No safe allowlist for trusted admin IP(s).</li>
  <li>Not validating each jail’s filter behavior against real traffic before hardening.</li>
</ul>

<p>With reverse proxy/auth setups (Traefik + Authentik), you can generate noisy 401/403/404 patterns while testing. If your scanner/auth jails are strict, self-ban becomes very easy.</p>

<h2 id="safer-configuration-i-use-now">Safer configuration I use now</h2>

<p>I moved to explicit, safer defaults and made non-SSH jail thresholds less trigger-happy while testing.</p>

<p><code class="language-plaintext highlighter-rouge">/etc/fail2ban/jail.local</code>:</p>

<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[DEFAULT]</span>
<span class="c"># Always keep your trusted admin/VPN IPs here.
</span><span class="py">ignoreip</span> <span class="p">=</span> <span class="s">127.0.0.1/8 ::1 &lt;MY_PUBLIC_IP&gt; &lt;VPN_IP&gt;</span>
<span class="py">bantime</span> <span class="p">=</span> <span class="s">1h</span>
<span class="py">findtime</span> <span class="p">=</span> <span class="s">10m</span>
<span class="py">maxretry</span> <span class="p">=</span> <span class="s">5</span>

<span class="nn">[sshd]</span>
<span class="py">enabled</span> <span class="p">=</span> <span class="s">true</span>
<span class="py">port</span> <span class="p">=</span> <span class="s">ssh</span>
<span class="py">backend</span> <span class="p">=</span> <span class="s">systemd</span>
<span class="py">logpath</span> <span class="p">=</span> <span class="s">/var/log/auth.log</span>
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">/etc/fail2ban/jail.d/custom-proxy.conf</code> (example):</p>

<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[custom-proxy-jail]</span>
<span class="py">enabled</span> <span class="p">=</span> <span class="s">true</span>
<span class="py">port</span> <span class="p">=</span> <span class="s">http,https</span>
<span class="py">filter</span> <span class="p">=</span> <span class="s">&lt;custom_filter_name&gt;</span>
<span class="py">logpath</span> <span class="p">=</span> <span class="s">&lt;proxy_access_log_path&gt;</span>
<span class="py">maxretry</span> <span class="p">=</span> <span class="s">&lt;tuned_value&gt;</span>
<span class="py">findtime</span> <span class="p">=</span> <span class="s">&lt;tuned_window&gt;</span>
<span class="py">bantime</span> <span class="p">=</span> <span class="s">&lt;tuned_duration&gt;</span>
<span class="py">action</span> <span class="p">=</span> <span class="s">iptables-multiport[name=custom-proxy-jail, port="80,443"]</span>
</code></pre></div></div>

<p>For RHEL-like systems, use <code class="language-plaintext highlighter-rouge">/var/log/secure</code> for <code class="language-plaintext highlighter-rouge">logpath</code>.</p>

<p>Then apply and re-check:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>systemctl restart fail2ban
<span class="nb">sudo </span>fail2ban-client status
<span class="nb">sudo </span>fail2ban-client status sshd
</code></pre></div></div>

<h2 id="extra-safety-practices">Extra safety practices</h2>

<p>Before changing ban rules now, I do this checklist:</p>

<ol>
  <li>Keep provider console access ready (do not skip this).</li>
  <li>Confirm <code class="language-plaintext highlighter-rouge">ignoreip</code> contains at least one trusted path back in.</li>
  <li>Test from a different IP/session before closing active shell.</li>
  <li>Tighten settings gradually, not all at once.</li>
  <li>Monitor jails after each config change.</li>
</ol>

<h2 id="practical-rollback-if-locked-out-again">Practical rollback if locked out again</h2>

<p>From provider console:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Temporary emergency disable</span>
<span class="nb">sudo </span>systemctl stop fail2ban

<span class="c"># Or unban from all jails in one command</span>
<span class="nb">sudo </span>fail2ban-client unban &lt;MY_PUBLIC_IP&gt;

<span class="c"># Or only unban specific jails</span>
<span class="nb">sudo </span>fail2ban-client <span class="nb">set </span>sshd unbanip &lt;MY_PUBLIC_IP&gt;
<span class="nb">sudo </span>fail2ban-client <span class="nb">set</span> &lt;proxy_or_auth_jail&gt; unbanip &lt;MY_PUBLIC_IP&gt;

<span class="c"># Fix config</span>
<span class="nb">sudo </span>editor /etc/fail2ban/jail.local

<span class="c"># Start again and verify</span>
<span class="nb">sudo </span>systemctl start fail2ban
<span class="nb">sudo </span>fail2ban-client status sshd
<span class="nb">sudo </span>fail2ban-client status &lt;proxy_or_auth_jail&gt;
</code></pre></div></div>

<h2 id="lessons-learned">Lessons learned</h2>

<ul>
  <li>Fail2ban is effective, but easy to misconfigure when hardening quickly.</li>
  <li>Reverse proxy + SSO testing can look like scanner behavior to Fail2ban filters.</li>
  <li>Always keep an out-of-band recovery path (provider console).</li>
  <li>Add trusted IPs before tightening bans.</li>
  <li>Keep an unban workflow/script ready before enabling stricter jails.</li>
</ul>

<h2 id="related-til">Related TIL</h2>

<ul>
  <li><a href="/today-i-learned/security/fail2ban-for-ssh/">Fail2ban for SSH</a></li>
</ul>

<h2 id="final-takeaway">Final takeaway</h2>

<p>Security controls should protect you from attackers, not block you from your own server.</p>

<p>Fail2ban is still worth using, but deploy it like a production change: safe defaults, staged tuning, and a guaranteed recovery path.</p>]]></content><author><name>Arshad Mehmood</name></author><category term="DevOps" /><category term="fail2ban" /><category term="security" /><category term="ssh" /><category term="traefik" /><category term="authentik" /><category term="linux" /><summary type="html"><![CDATA[A real incident report: I got my own IP banned after enabling Fail2ban with Traefik and Authentik, lost SSH access, recovered via provider console, and fixed my configuration.]]></summary></entry><entry><title type="html">We All Played Destiny, but Bungie Played Us All</title><link href="https://arshadmehmood.com/personal/we-all-played-destiny-but-bungie-played-us-all/" rel="alternate" type="text/html" title="We All Played Destiny, but Bungie Played Us All" /><published>2026-03-27T00:00:00+00:00</published><updated>2026-03-27T00:00:00+00:00</updated><id>https://arshadmehmood.com/personal/we-all-played-destiny-but-bungie-played-us-all</id><content type="html" xml:base="https://arshadmehmood.com/personal/we-all-played-destiny-but-bungie-played-us-all/"><![CDATA[<p>I have played <em>Destiny 2</em> for more than seven years. I have over 1,000 Grandmaster Nightfall clears. Around 70% of those I did solo. My vault and characters are packed with masterworked gear. I sit on piles of cores, shards, and endgame materials. On Stadia, I was consistently one of the top players because <em>Destiny 2</em> was not just <em>a</em> game to me.</p>

<p>It was <em>the</em> game.</p>

<p>For years, when people asked what I played, my answer was simple:</p>

<blockquote>
  <p><strong>I only play one game: Destiny 2.</strong></p>
</blockquote>

<p>That is exactly why I feel qualified to say this: <strong>Bungie killed its golden goose.</strong></p>

<p>This is not a neutral post. It is a goodbye letter from a player who gave <em>Destiny 2</em> years of time, loyalty, money, and patience. And the reason I am done is not just that the game changed.</p>

<p>It is that Bungie changed.</p>

<h2 id="from-passion-project-to-extraction-machine">From Passion Project to Extraction Machine</h2>

<p>There was a time when Bungie seemed to genuinely care about its players. The game had problems, sure, but there was still a sense that the developers wanted to build something special and lasting. There was a soul there. You could feel it in the world design, in the raids, in the gunplay, in the music, and in the moments where the game actually respected your time.</p>

<p>But as the years went on, that version of Bungie got buried under corporate thinking, monetization layers, and cynical live-service decision-making. What started as a beloved shared-world shooter slowly turned into a machine built to extract more money while giving less back.</p>

<p>And that shift did not happen by accident. It happened step by step, update by update, expansion by expansion, excuse by excuse.</p>

<h2 id="they-blamed-activision-then-proved-they-were-the-problem">They Blamed Activision, Then Proved They Were the Problem</h2>

<p>For a long time, a lot of the player base blamed Activision for everything wrong with <em>Destiny</em>. The aggressive monetization, the business-first design, the compromises, the artificial grind: people assumed that was all Activision’s influence.</p>

<p>Then Bungie split from Activision.</p>

<p>That should have been the moment everything improved. If Activision had truly been the source of the rot, Bungie’s independence should have led to a more honest, more player-friendly <em>Destiny</em>. Instead, many of the worst decisions either stayed, intensified, or evolved into something even more shameless.</p>

<p>That is why I believe Bungie used Activision as cover. They let players believe the publisher was holding them back, and once the publisher was gone, the same patterns remained. Same monetization instincts. Same manipulative design. Same disrespect for players’ time and money.</p>

<p>That excuse died the moment Bungie stood on its own and kept doing the same thing.</p>

<h2 id="destiny-became-microtransaction-hell">Destiny Became Microtransaction Hell</h2>

<p>This is where the betrayal became impossible to ignore.</p>

<p><em>Destiny 2</em> stopped feeling like one premium hobby and started feeling like a game designed around toll booths. Everything became segmented, monetized, and repackaged.</p>

<p>You do not just buy an expansion anymore. You buy the DLC. Then the season. Then the dungeon. Then the cosmetics. Then the event pass. Then the transmog shortcuts. Then whatever other Silver-based nonsense they have cooked up.</p>

<p>At some point it became absurd. A major expansion package can cost close to 100€, and somehow that still does not feel like the full product. You are still expected to keep paying on top of that. For a game that already charges premium expansion prices, this level of layered monetization is indefensible.</p>

<p>They turned <em>Destiny 2</em> into a storefront disguised as a live-service game.</p>

<h2 id="the-train-station-mindset-said-everything">The “Train Station” Mindset Said Everything</h2>

<p>What makes this worse is that Bungie has openly revealed the mindset behind all of this.</p>

<p>At a developer conference, Bungie used the analogy that they were not just building a train, they were building a train station: a system designed to keep the live-service machine going. Another Bungie presentation included the now infamous idea that “overdelivery” is dangerous, because if you give players too much, they will expect that level again.</p>

<p>That explains so much.</p>

<p>For years, players felt like Bungie was deliberately holding back, stretching content, rationing quality, and constantly managing expectations downward. We were told not to expect too much. We were trained to celebrate the return of things that should never have left. We were conditioned to accept less and call it sustainability.</p>

<p>That “train station” mentality is exactly how <em>Destiny 2</em> started to feel: not like a world built for players, but like an infrastructure built to keep extracting engagement and money.</p>

<h2 id="they-sold-power-first-then-balanced-it-later">They Sold Power First, Then Balanced It Later</h2>

<p>One of the most toxic patterns in <em>Destiny 2</em> has been the repeated release of overtuned abilities, subclasses, and weapons that just happen to be tied to paid content, only for them to get nerfed later after enough people have bought in.</p>

<p>Maybe Bungie would call that a balancing issue. Maybe they would say live-service tuning is complicated. Fine. But players are not stupid. When the newest paid content repeatedly launches too strong and only gets brought back in line after the purchases have gone through, people notice.</p>

<p>Stasis was one of the biggest examples. There have been others. New exotics, subclass interactions, artifact perks, seasonal weapon dominance: again and again, Bungie created the impression that buying in early meant getting access to the strongest toys before the inevitable tuning pass.</p>

<p>Even when intent cannot be proven, the pattern destroys trust. And once trust is gone, every “oops, this launched too strong” starts sounding less like a mistake and more like a business tactic.</p>

<h2 id="sunsetting-was-a-betrayal-of-player-time">Sunsetting Was a Betrayal of Player Time</h2>

<p>Sunsetting was one of the worst decisions Bungie ever made.</p>

<p>Players spent years grinding for god rolls, perfect builds, masterworked gear, and weapons tied to difficult content. Then Bungie basically told us that huge portions of that loot would be left behind, invalidated, or made irrelevant in power-enabled content.</p>

<p>The official excuse was sandbox health, power creep, and making room for new rewards. But to many of us, it looked like a lazy, brute-force solution to weapons Bungie could not properly balance. Instead of dealing with their own design problems intelligently, they chose to devalue years of player investment.</p>

<p>That is the kind of decision that permanently damages a game’s relationship with its most loyal community. And the fact that Bungie later walked back major parts of that philosophy only made the original decision look even more indefensible.</p>

<p>They wasted players’ time, then quietly admitted the backlash was justified.</p>

<h2 id="vaulting-paid-content-crossed-the-line">Vaulting Paid Content Crossed the Line</h2>

<p>Sunsetting was bad. Vaulting was worse.</p>

<p>Bungie removed entire destinations, campaigns, and DLC content from a game people had already paid for. Not metaphorically. Literally. Content was bought, then taken away. Expansions and planets that were part of the product disappeared into the so-called vault.</p>

<p>That should have been unacceptable from day one.</p>

<p>It sent a horrible message: in <em>Destiny 2</em>, your purchases are temporary, but Bungie’s right to keep selling you new things is permanent.</p>

<p>And then, to make it even worse, Bungie spent years pulling old ideas, locations, weapons, and activities back out of the vault, repainting or reskinning them, and selling nostalgia back to the same player base that had already paid once. That is not clever live-service content management. That is recycling old value after deleting it and pretending it is generosity when it returns.</p>

<p>The DLCs I bought are not even there anymore. That alone tells you everything you need to know.</p>

<h2 id="lightfall-felt-like-filler-not-a-real-expansion">Lightfall Felt Like Filler, Not a Real Expansion</h2>

<p>After <em>The Witch Queen</em>, expectations were high. Bungie had proven they could still deliver something focused, meaningful, and high quality when they wanted to.</p>

<p>Then came <em>Lightfall</em>.</p>

<p>To me, <em>Lightfall</em> was not a proper chapter in Destiny’s saga. It was filler. It felt like an expansion created to stretch the roadmap, buy time, and squeeze more money out of a player base that was already deeply invested and emotionally locked in.</p>

<p>The story felt disconnected. The tone felt off. The pacing was strange. The stakes were supposedly massive, yet so much of it felt hollow and half-baked. Instead of feeling like the natural next step, it felt like a detour that existed because Bungie needed another stopgap release before getting to the actual conclusion.</p>

<p>It did not fit properly because it was never built with the same integrity as the expansions around it. It felt like padding.</p>

<h2 id="strand-should-have-been-in-the-witch-queen">Strand Should Have Been in The Witch Queen</h2>

<p>I do not care how many times Bungie denies it: I still believe Strand was originally meant to be part of <em>The Witch Queen</em>.</p>

<p>There are too many clues. The visual language fits. The themes line up. The psychic, thread-like, reality-bending nature of Strand feels like it belongs far more naturally with Savathun, Deepsight, unraveling hidden truths, and the mystery-heavy tone of <em>The Witch Queen</em> than with the neon chaos of <em>Lightfall</em>.</p>

<p>To me, Strand being moved out and <em>Lightfall</em> being stretched into its own expansion looks like a deliberate attempt to split content apart and monetize it separately.</p>

<p>Maybe Bungie says otherwise. That is their line. I do not buy it.</p>

<p>Players are not crazy for looking at the timeline and concluding that <em>Lightfall</em> existed to squeeze out one more big expansion sale.</p>

<h2 id="they-kept-adding-friction-everywhere">They Kept Adding Friction Everywhere</h2>

<p>Beyond the flashy controversies, Bungie also spent years slowly choking the game with friction.</p>

<p>Too many currencies. Too many systems. Too many rotating passes. Too many convoluted purchase models. Too many layers between the player and the content they actually want to play.</p>

<p>The onboarding for new players remained awful. Ritual content stagnated. Important systems were constantly reworked, often without truly improving the core experience. Veteran players had to keep re-learning systems that did not need to be this messy in the first place, while new players were thrown into a confusing maze.</p>

<p>It increasingly felt like Bungie was designing around monetization, engagement metrics, and retention loops first, and fun second.</p>

<h2 id="vault-space-was-another-obvious-lie">Vault Space Was Another Obvious Lie</h2>

<p>And yes, I am going to say it plainly: the excuses about vault space never sat right with me.</p>

<p>As a developer myself, Bungie’s logic around storage always sounded like bullshit. For years players were told, directly or indirectly, that vault limitations were technical, difficult, or necessary. Meanwhile the game kept vomiting out more guns, more armor, more perks, more crafted variations, more curated rolls, more event items, and more reasons to hold onto gear.</p>

<p>The numbers simply did not match the reality of how they designed the loot game.</p>

<p>Players kept asking for meaningful vault improvements while Bungie kept acting like this was some impossible engineering mountain. Meanwhile their CEO was out buying supercars and the people keeping the game alive were still being told to juggle inventories in a loot-based MMO-lite built entirely around collecting things.</p>

<p>That is why so many players stopped believing them. Not because storage is magic, but because the excuses always seemed to appear when the requested quality-of-life improvement did not directly generate revenue.</p>

<h2 id="bungie-neglected-destiny-while-chasing-marathon">Bungie Neglected Destiny While Chasing Marathon</h2>

<p>This is another reason the bitterness runs so deep.</p>

<p>For years, Destiny players supported Bungie through all the ups and downs. We kept the lights on. We bought the expansions. We bought the seasons. We tolerated the monetization. We stuck around through the dry spells. We defended the game to others. We gave them loyalty most studios would kill for.</p>

<p>And what did Bungie do with that support?</p>

<p>They used <em>Destiny</em> as a money-printing machine while shifting focus toward <em>Marathon</em>.</p>

<p>Instead of reinvesting enough of that money into strengthening <em>Destiny</em>, improving the core game, respecting longtime players, and building a healthier future for the franchise that fed them for years, they seemed more interested in funding the next project.</p>

<p>So yes, I will say it openly: it makes me happy that <em>Marathon</em> is not doing well.</p>

<p>That may sound harsh, but after watching Bungie milk <em>Destiny</em> while treating it more and more like a revenue source rather than a world worth protecting, I do not have sympathy left. They used their golden goose to fund a new dream, and they damaged the old one in the process.</p>

<h2 id="even-basic-respect-for-artists-became-a-problem">Even Basic Respect for Artists Became a Problem</h2>

<p>For a studio that is treated like AAA royalty, Bungie has repeatedly embarrassed itself with incidents involving stolen or unauthorized art and concepts.</p>

<p>That matters.</p>

<p>Because this is not just about bugs, balancing mistakes, or bad monetization. It is about professionalism and ethics. When a studio keeps ending up in situations where indie artists’ work appears to have been used without proper credit, permission, or compensation, it reinforces the image of a company that feels entitled to take from smaller creators while hiding behind corporate statements afterward.</p>

<p>That kind of behavior fits the broader pattern: exploit value wherever possible, apologize when caught, and move on.</p>

<p>For a company of Bungie’s size and status, that is pathetic.</p>

<h2 id="bungie-is-called-aaa-but-too-often-they-act-lazy">Bungie Is Called AAA, but Too Often They Act Lazy</h2>

<p>Yes, Bungie is recognized as a AAA studio. But from the player side, too often they do not act like one.</p>

<p>A real top-tier studio does not repeatedly hide behind weak excuses, recycle old content as new value, remove paid content, drag out basic quality-of-life improvements, ship half-baked expansions, and constantly push monetization deeper into every corner of the game while asking for patience and trust.</p>

<p>A real AAA studio does not treat loyalty like something to be farmed.</p>

<p>Too often Bungie has felt less like a world-class developer and more like a studio coasting on old reputation, excellent gunplay, and the emotional investment of a trapped player base.</p>

<h2 id="the-portal-was-the-final-straw-for-me">The Portal Was the Final Straw for Me</h2>

<p>Recently they added the Portal, and honestly: what the hell even is that?</p>

<p>I only used to play Grandmaster Nightfalls. That was my thing. That was the part of the game I cared about most. That was the content loop I kept coming back for. I did not need some abstracted “portal” structure or whatever new activity framework Bungie wants players to funnel through.</p>

<p>And now Grandmaster Nightfalls, as they used to exist in the way I cared about them, are effectively gone or absorbed into a system I have no interest in engaging with.</p>

<p>I do not have time for their Portal bullshit.</p>

<p>I do not want another layer. I do not want another UI experiment. I do not want another system designed to repackage activities into a new engagement funnel. I wanted to log in and play the hardest PvE content in the game the way I had for years.</p>

<p>That was enough for me. Bungie removed the thing I loved and replaced it with something that feels like corporate product design.</p>

<p>That was my breaking point.</p>

<h2 id="this-is-not-coming-from-a-casual-player">This Is Not Coming From a Casual Player</h2>

<p>And that matters.</p>

<p>I am not writing this as someone who dipped in for a few expansions and got bored. I am not writing this as someone who watched a few YouTube videos and decided to pile on. I am writing this as someone who lived in this game.</p>

<p>Over seven years.<br />
Over 1,000 Grandmaster clears.<br />
Roughly 70% of them solo.<br />
A full vault.<br />
Full characters.<br />
Masterworked gear everywhere.<br />
70,000 cores.<br />
Ascendant shards stacked in the postmaster.<br />
Top-tier commitment.<br />
Top Stadia player.<br />
One-game player.</p>

<p>I know what <em>Destiny 2</em> was. I know what it became. And I know what it feels like when a game you truly loved stops respecting the people who kept it alive.</p>

<p>So yes, I have earned the right to say this.</p>

<h2 id="the-player-counts-tell-their-own-story">The Player Counts Tell Their Own Story</h2>

<p>When I see the <em>Destiny 2</em> player counts now, I do not feel shocked anymore. I feel validated.</p>

<p>It makes me happy that so many people have moved on. Not because I wanted <em>Destiny</em> to die, but because I wanted players to stop falling for Bungie’s traps. For too long, this company relied on loyalty, habit, sunk cost, and emotional attachment to keep people paying into a system that was giving less and less back.</p>

<p>Seeing more players finally walk away tells me I was not imagining it. A lot of people saw the same thing. A lot of people got tired of being manipulated, monetized, and ignored.</p>

<p>That is not tragedy. That is clarity.</p>

<h2 id="bungie-killed-its-golden-goose">Bungie Killed Its Golden Goose</h2>

<p>This is the saddest part of all.</p>

<p><em>Destiny 2</em> did not collapse because the gunplay stopped being incredible. It did not fail because the art team lost their talent. It did not fail because the core fantasy was weak. The foundation was always there. The potential was always there.</p>

<p>Bungie killed its golden goose because it kept choosing short-term monetization, player manipulation, and executive priorities over long-term trust.</p>

<p>They took one of the most mechanically satisfying shooters ever made and buried it under greed.<br />
They took a loyal community and treated it like a resource to be mined.<br />
They took years of player goodwill and burned through it for expansions, seasons, dungeon keys, recycled content, and a future that no longer seemed to care about the players who built that success in the first place.</p>

<p>Many hardcore <em>Destiny</em> players supported Bungie for years.</p>

<p>And Bungie betrayed everyone.</p>

<h2 id="goodbye-destiny">Goodbye, Destiny</h2>

<p>It is enough for me.</p>

<p>I do not have the energy anymore to keep justifying Bungie’s decisions, hoping the next expansion fixes things, or pretending the pattern is not real. I do not want to get dragged into more systems, more monetization layers, more repackaged activities, more excuses, or more disappointment.</p>

<p>I would not give Bungie a single penny ever again.</p>

<p>And honestly, I do not think anyone else should either.</p>

<p>This is my goodbye, not just to a game, but to the illusion that Bungie still deserves the kind of loyalty players like me gave them for years.</p>

<p>We stayed.<br />
We defended them.<br />
We paid.<br />
We grinded.<br />
We believed.</p>

<p>And in the end, they played us.</p>

<h2 id="final-thought">Final Thought</h2>

<p><strong>We all played Destiny, but Bungie played us all.</strong></p>]]></content><author><name>Arshad Mehmood</name></author><category term="Personal" /><category term="Destiny 2" /><category term="Bungie" /><category term="Gaming" /><category term="Live Service" /><category term="Microtransactions" /><category term="Opinion" /><summary type="html"><![CDATA[After more than seven years, 1,000+ Grandmaster Nightfall clears, and thousands of hours in Destiny 2, this is my goodbye letter to a game Bungie slowly turned into an extraction machine.]]></summary></entry><entry><title type="html">ChatGPT is Secretly Sharing Conversations Within Projects: A Privacy Investigation</title><link href="https://arshadmehmood.com/software/chatgpt-secretly-sharing-conversations-within-projects-privacy-investigation/" rel="alternate" type="text/html" title="ChatGPT is Secretly Sharing Conversations Within Projects: A Privacy Investigation" /><published>2025-10-23T00:00:00+00:00</published><updated>2025-10-23T00:00:00+00:00</updated><id>https://arshadmehmood.com/software/chatgpt-secretly-sharing-conversations-within-projects-privacy-investigation</id><content type="html" xml:base="https://arshadmehmood.com/software/chatgpt-secretly-sharing-conversations-within-projects-privacy-investigation/"><![CDATA[<h2 id="introduction">Introduction</h2>

<p>Have you ever wondered if ChatGPT remembers things you’ve told it in previous conversations, even when you’ve disabled memory? I’ve been suspicious about this for quite some time, particularly regarding ChatGPT’s <strong>Projects</strong> feature. After conducting a systematic test, I discovered something concerning: <strong>ChatGPT is indeed sharing conversation context between separate chats within the same project</strong>, even when memory settings are explicitly turned off.</p>

<p>This investigation reveals a significant privacy implication that OpenAI doesn’t clearly communicate to users. Let me walk you through my findings.</p>

<h2 id="the-discovery-memory-sharing-in-projects">The Discovery: Memory Sharing in Projects</h2>

<h3 id="setting-the-stage-memory-disabled">Setting the Stage: Memory Disabled</h3>

<p>Before conducting my test, I verified that my ChatGPT memory settings were completely disabled. This is crucial because it eliminates the obvious explanation for any information retention.</p>

<p class="text-center"><img src="/assets/images/posts/Chatgpt_memory_off.PNG" alt="ChatGPT Memory Settings Disabled" class="align-center" style="max-width: 70%; height: auto;" />
<em>Memory settings clearly set to “Off”</em></p>

<h3 id="the-experiment-testing-information-persistence">The Experiment: Testing Information Persistence</h3>

<h4 id="step-1-initial-query-in-a-fresh-chat">Step 1: Initial Query in a Fresh Chat</h4>

<p>I started by creating a new chat within a project and asking ChatGPT if it knew my belly size. As expected, it had no prior knowledge of this information.</p>

<p class="text-center"><img src="/assets/images/posts/Chatgpt_fresh.PNG" alt="Fresh Chat - No Prior Knowledge" class="align-center" style="max-width: 70%; height: auto;" />
<em>ChatGPT correctly states it has no knowledge of personal information</em></p>

<h4 id="step-2-sharing-personal-information-in-a-new-chat">Step 2: Sharing Personal Information in a New Chat</h4>

<p>I then created a completely separate chat within the same project and shared personal information about my belly size. This was done to test if this information would somehow persist across different conversations.</p>

<p class="text-center"><img src="/assets/images/posts/Chatgpt_sharing.PNG" alt="Sharing Personal Information" class="align-center" style="max-width: 70%; height: auto;" />
<em>Sharing personal information in a separate chat within the same project</em></p>

<h4 id="step-3-the-shocking-discovery">Step 3: The Shocking Discovery</h4>

<p>When I returned to the first chat and asked the same question again, ChatGPT suddenly “remembered” the information I had shared in the completely separate conversation!</p>

<p class="text-center"><img src="/assets/images/posts/Chatgpt_again.PNG" alt="ChatGPT Remembers Across Chats" class="align-center" style="max-width: 70%; height: auto;" />
<em>ChatGPT now has knowledge it shouldn’t possess based on disabled memory settings</em></p>

<h2 id="confirming-the-scope">Confirming the Scope</h2>

<h3 id="project-specific-behavior">Project-Specific Behavior</h3>

<p>To understand the scope of this behavior, I verified that only two chats existed in my test project, confirming that the information sharing was contained within the project boundary.</p>

<p class="text-center"><img src="/assets/images/posts/Chatgpt_chats.PNG" alt="Only Two Chats in Project" class="align-center" style="max-width: 70%; height: auto;" />
<em>Confirmation that only two chats existed in the test project</em></p>

<h3 id="testing-outside-projects">Testing Outside Projects</h3>

<p>I conducted the same test in the main ChatGPT interface (outside of any project) and confirmed that this memory-sharing behavior does <strong>not</strong> occur in regular chats.</p>

<p class="text-center"><img src="/assets/images/posts/Chatgpt_main.PNG" alt="No Memory Sharing in Main Chats" class="align-center" style="max-width: 70%; height: auto;" />
<em>Regular chats outside projects don’t exhibit this behavior</em></p>

<h2 id="the-contradiction-what-chatgpt-claims-vs-reality">The Contradiction: What ChatGPT Claims vs. Reality</h2>

<h3 id="chatgpts-response-to-direct-questioning">ChatGPT’s Response to Direct Questioning</h3>

<p>When directly asked about this behavior, ChatGPT denies saving or sharing information between chats within projects:</p>

<p class="text-center"><img src="/assets/images/posts/Chatgpt_saves_memory.PNG" alt="ChatGPT's Denial" class="align-center" style="max-width: 70%; height: auto;" />
<em>ChatGPT claims it doesn’t save or share information between chats</em></p>

<p>This creates a concerning contradiction between what ChatGPT claims about its behavior and what actually happens in practice.</p>

<h2 id="limited-documentation-and-transparency">Limited Documentation and Transparency</h2>

<h3 id="scarce-official-information">Scarce Official Information</h3>

<p>The only reference I could find acknowledging this behavior was in a third-party article:
<a href="https://www.techradar.com/computing/artificial-intelligence/openai-has-upgraded-chatgpts-projects-feature-and-i-find-it-makes-working-way-more-efficient">OpenAI has upgraded ChatGPT’s Projects feature, and I find it makes working way more efficient</a></p>

<h3 id="missing-from-official-documentation">Missing from Official Documentation</h3>

<p>Notably, OpenAI’s official website and documentation do not clearly explain this behavior. There are no settings to:</p>
<ul>
  <li>Control memory sharing within projects</li>
  <li>Disable context sharing between chats in a project</li>
  <li>Manage privacy preferences for project-level information retention</li>
</ul>

<h2 id="privacy-implications-and-concerns">Privacy Implications and Concerns</h2>

<h3 id="what-this-means-for-users">What This Means for Users</h3>

<p>This discovery has several important implications:</p>

<ol>
  <li>
    <p><strong>Hidden Data Persistence</strong>: Information you share in one chat within a project persists across all other chats in that project, regardless of your memory settings.</p>
  </li>
  <li>
    <p><strong>Lack of User Control</strong>: There’s no clear way to disable this behavior or manage what information is shared within a project.</p>
  </li>
  <li>
    <p><strong>Documentation Gap</strong>: The behavior isn’t clearly documented, leaving users unaware of how their data is being handled.</p>
  </li>
  <li>
    <p><strong>False Sense of Privacy</strong>: Users who disable memory settings may believe their conversations are isolated, when they’re actually not within projects.</p>
  </li>
</ol>

<h3 id="potential-risks">Potential Risks</h3>

<ul>
  <li><strong>Unintended Information Disclosure</strong>: Sensitive information shared in one project chat becomes available in all other chats within that project</li>
  <li><strong>Context Contamination</strong>: Previous conversations can influence AI responses in ways users don’t expect</li>
  <li><strong>Privacy Confusion</strong>: Users may not understand the true scope of information retention</li>
</ul>

<h2 id="recommendations">Recommendations</h2>

<h3 id="for-users">For Users</h3>

<ol>
  <li><strong>Be Aware</strong>: Understand that information shared in any chat within a ChatGPT project persists across all chats in that project</li>
  <li><strong>Project Hygiene</strong>: Consider creating separate projects for sensitive or unrelated topics</li>
  <li><strong>Regular Cleanup</strong>: Periodically review and clean up your ChatGPT projects</li>
  <li><strong>Verify Privacy Settings</strong>: Don’t rely solely on memory settings to control information persistence</li>
</ol>

<h3 id="for-openai">For OpenAI</h3>

<ol>
  <li><strong>Improve Documentation</strong>: Clearly explain how Projects handle information sharing</li>
  <li><strong>Add User Controls</strong>: Provide settings to control context sharing within projects</li>
  <li><strong>Enhance Transparency</strong>: Be more explicit about data handling practices</li>
  <li><strong>Fix Contradictory Responses</strong>: Ensure ChatGPT accurately describes its own behavior</li>
</ol>

<h2 id="conclusion">Conclusion</h2>

<p>This investigation reveals a significant gap between user expectations and ChatGPT’s actual behavior within Projects. While the feature may be designed to improve workflow efficiency by maintaining context, the lack of clear documentation and user control raises important privacy concerns.</p>

<p>The most troubling aspect is the contradiction between what ChatGPT claims about its behavior and what actually happens. Users deserve clear, accurate information about how their data is being handled, especially when privacy settings suggest certain protections are in place.</p>

<p>Until OpenAI provides better transparency and user controls, it’s important for users to be aware of this behavior and take appropriate precautions when sharing sensitive information in ChatGPT Projects.</p>

<p><strong>Have you noticed similar behavior in your ChatGPT usage? Share your experiences in the comments below.</strong></p>

<hr />

<p><em>This investigation was conducted on October 23, 2025, using ChatGPT’s Projects feature. Results may vary as OpenAI updates their platform.</em></p>]]></content><author><name>Arshad Mehmood</name></author><category term="Software" /><category term="Artificial Intelligence" /><category term="ChatGPT" /><category term="AI" /><category term="LLM" /><category term="Privacy" /><category term="OpenAI" /><category term="Data Sharing" /><summary type="html"><![CDATA[A detailed investigation revealing how ChatGPT's Projects feature shares conversation context between separate chats, despite users having memory settings disabled.]]></summary></entry><entry><title type="html">Avoiding Race Conditions in Go: A Practical Guide to sync/atomic</title><link href="https://arshadmehmood.com/development/avoiding-race-conditions-in-go/" rel="alternate" type="text/html" title="Avoiding Race Conditions in Go: A Practical Guide to sync/atomic" /><published>2025-09-05T00:00:00+00:00</published><updated>2025-09-05T00:00:00+00:00</updated><id>https://arshadmehmood.com/development/avoiding-race-conditions-in-go</id><content type="html" xml:base="https://arshadmehmood.com/development/avoiding-race-conditions-in-go/"><![CDATA[<p>As I continue my Go learning journey, I frequently encounter the power and challenges of concurrent programming. Goroutines make it incredibly easy to write parallel code, but they also introduce one of the most common and dangerous bugs in concurrent programming: <strong>race conditions</strong>.</p>

<p>Recently, I ran into this exact issue while building a tool to manage cloud instances. What seemed like simple counting logic turned into a debugging nightmare! In this post, I’ll walk you through the same scenario and show you how Go’s <code class="language-plaintext highlighter-rouge">sync/atomic</code> package saved the day.</p>

<h2 id="understanding-the-problem">Understanding the Problem</h2>

<p>Imagine you’re building a cloud management tool that needs to update multiple server instances concurrently. You want to count how many instances were successfully updated. Here’s what the initial implementation might look like:</p>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span>

<span class="k">import</span> <span class="p">(</span>
    <span class="s">"fmt"</span>
    <span class="s">"time"</span>
    <span class="s">"golang.org/x/sync/errgroup"</span>
<span class="p">)</span>

<span class="c">// Simulate updating an instance (for demonstration)</span>
<span class="k">func</span> <span class="n">updateInstance</span><span class="p">(</span><span class="n">instance</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
    <span class="c">// Simulate some work</span>
    <span class="n">time</span><span class="o">.</span><span class="n">Sleep</span><span class="p">(</span><span class="m">10</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Millisecond</span><span class="p">)</span>
    <span class="c">// Simulate occasional failures</span>
    <span class="k">if</span> <span class="n">instance</span> <span class="o">==</span> <span class="s">"instance-error"</span> <span class="p">{</span>
        <span class="k">return</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"failed to update %s"</span><span class="p">,</span> <span class="n">instance</span><span class="p">)</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="no">nil</span>
<span class="p">}</span>

<span class="k">func</span> <span class="n">updateInstances</span><span class="p">(</span><span class="n">instances</span> <span class="p">[]</span><span class="kt">string</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
    <span class="k">var</span> <span class="n">wg</span> <span class="n">errgroup</span><span class="o">.</span><span class="n">Group</span>
    <span class="n">updatedCount</span> <span class="o">:=</span> <span class="m">0</span>  <span class="c">// 🚨 This will cause problems!</span>
    
    <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">instance</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">instances</span> <span class="p">{</span>
        <span class="n">instance</span> <span class="o">:=</span> <span class="n">instance</span> <span class="c">// capture loop variable</span>
        <span class="n">wg</span><span class="o">.</span><span class="n">Go</span><span class="p">(</span><span class="k">func</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
            <span class="n">err</span> <span class="o">:=</span> <span class="n">updateInstance</span><span class="p">(</span><span class="n">instance</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="no">nil</span> <span class="p">{</span>
                <span class="n">updatedCount</span><span class="o">++</span>  <span class="c">// ❌ RACE CONDITION!</span>
            <span class="p">}</span>
            <span class="k">return</span> <span class="n">err</span>
        <span class="p">})</span>
    <span class="p">}</span>
    
    <span class="k">if</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">wg</span><span class="o">.</span><span class="n">Wait</span><span class="p">();</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
        <span class="k">return</span> <span class="n">err</span>
    <span class="p">}</span>
    
    <span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"Successfully updated %d instances</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">updatedCount</span><span class="p">)</span>
    <span class="k">return</span> <span class="no">nil</span>
<span class="p">}</span>
</code></pre></div></div>

<p>This code looks reasonable, but it contains a subtle and dangerous bug. Can you spot it?</p>

<h2 id="the-aha-moment-understanding-atomic-operations">The “Aha!” Moment: Understanding Atomic Operations</h2>

<p>The <code class="language-plaintext highlighter-rouge">sync/atomic</code> package provides <strong>atomic operations</strong> - think of them like transactions at an ATM. When you withdraw money, either the entire operation completes (your balance updates AND you get cash), or nothing happens at all. There’s no in-between state where the money disappears!</p>

<h2 id="the-race-condition-explained">The Race Condition Explained</h2>

<p>In our example above, we have a classic race condition. Let’s break down what’s happening:</p>

<p><strong>The problematic code:</strong></p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">var</span> <span class="n">wg</span> <span class="n">errgroup</span><span class="o">.</span><span class="n">Group</span>
<span class="n">updatedCount</span> <span class="o">:=</span> <span class="m">0</span>  <span class="c">// Shared variable</span>

<span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">instance</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">instances</span> <span class="p">{</span>
    <span class="n">wg</span><span class="o">.</span><span class="n">Go</span><span class="p">(</span><span class="k">func</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
        <span class="c">// Multiple goroutines running concurrently</span>
        <span class="n">err</span> <span class="o">:=</span> <span class="n">updateInstance</span><span class="p">(</span><span class="n">instance</span><span class="p">)</span>
        <span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="no">nil</span> <span class="p">{</span>
            <span class="n">updatedCount</span><span class="o">++</span>  <span class="c">// ❌ RACE CONDITION!</span>
        <span class="p">}</span>
        <span class="k">return</span> <span class="n">err</span>
    <span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>

<h3 id="why-this-is-dangerous">Why This is Dangerous</h3>

<p>The problem lies in the innocent-looking line <code class="language-plaintext highlighter-rouge">updatedCount++</code>. This operation seems atomic, but it’s actually <strong>three separate operations</strong> at the CPU level:</p>

<ol>
  <li><strong>Read</strong> the current value of <code class="language-plaintext highlighter-rouge">updatedCount</code></li>
  <li><strong>Add</strong> 1 to that value</li>
  <li><strong>Write</strong> the new value back to <code class="language-plaintext highlighter-rouge">updatedCount</code></li>
</ol>

<h3 id="the-race-condition-in-action">The Race Condition in Action</h3>

<p>Here’s what can happen when multiple goroutines execute this code simultaneously:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Timeline: Two goroutines (A and B) trying to increment updatedCount

Initial value: updatedCount = 5

Goroutine A: Read updatedCount (5)
Goroutine B: Read updatedCount (5)    // Still 5! B doesn't see A's changes yet
Goroutine A: Calculate 5 + 1 = 6
Goroutine B: Calculate 5 + 1 = 6      // Also calculating from 5!
Goroutine A: Write 6 back
Goroutine B: Write 6 back             // Overwrites A's work!

Final value: updatedCount = 6 (should be 7!)
</code></pre></div></div>

<p><strong>Result: We lost one increment!</strong> If you had 100 successful updates, you might see anywhere from 1 to 100 as the final count, depending on the timing of the race conditions.</p>

<h2 id="the-solution-atomic-operations">The Solution: Atomic Operations</h2>

<p>Now let’s fix our code using atomic operations:</p>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span>

<span class="k">import</span> <span class="p">(</span>
    <span class="s">"fmt"</span>
    <span class="s">"sync/atomic"</span>
    <span class="s">"time"</span>
    <span class="s">"golang.org/x/sync/errgroup"</span>
<span class="p">)</span>

<span class="c">// Simulate updating an instance (for demonstration)</span>
<span class="k">func</span> <span class="n">updateInstance</span><span class="p">(</span><span class="n">instance</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
    <span class="c">// Simulate some work</span>
    <span class="n">time</span><span class="o">.</span><span class="n">Sleep</span><span class="p">(</span><span class="m">10</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Millisecond</span><span class="p">)</span>
    <span class="c">// Simulate occasional failures</span>
    <span class="k">if</span> <span class="n">instance</span> <span class="o">==</span> <span class="s">"instance-error"</span> <span class="p">{</span>
        <span class="k">return</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"failed to update %s"</span><span class="p">,</span> <span class="n">instance</span><span class="p">)</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="no">nil</span>
<span class="p">}</span>

<span class="k">func</span> <span class="n">updateInstancesSafely</span><span class="p">(</span><span class="n">instances</span> <span class="p">[]</span><span class="kt">string</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
    <span class="k">var</span> <span class="n">wg</span> <span class="n">errgroup</span><span class="o">.</span><span class="n">Group</span>
    <span class="k">var</span> <span class="n">updatedCount</span> <span class="kt">int64</span>  <span class="c">// ✅ Must be int64 for atomic operations</span>

    <span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">instance</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">instances</span> <span class="p">{</span>
        <span class="n">instance</span> <span class="o">:=</span> <span class="n">instance</span> <span class="c">// capture loop variable</span>
        <span class="n">wg</span><span class="o">.</span><span class="n">Go</span><span class="p">(</span><span class="k">func</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
            <span class="n">err</span> <span class="o">:=</span> <span class="n">updateInstance</span><span class="p">(</span><span class="n">instance</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="no">nil</span> <span class="p">{</span>
                <span class="n">atomic</span><span class="o">.</span><span class="n">AddInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="n">updatedCount</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span>  <span class="c">// ✅ THREAD-SAFE!</span>
            <span class="p">}</span>
            <span class="k">return</span> <span class="n">err</span>
        <span class="p">})</span>
    <span class="p">}</span>

    <span class="k">if</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">wg</span><span class="o">.</span><span class="n">Wait</span><span class="p">();</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
        <span class="k">return</span> <span class="n">err</span>
    <span class="p">}</span>

    <span class="c">// Read the final value atomically</span>
    <span class="n">finalCount</span> <span class="o">:=</span> <span class="n">atomic</span><span class="o">.</span><span class="n">LoadInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="n">updatedCount</span><span class="p">)</span>
    <span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"Successfully updated %d instances</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">finalCount</span><span class="p">)</span>
    <span class="k">return</span> <span class="no">nil</span>
<span class="p">}</span>
</code></pre></div></div>

<h3 id="key-changes">Key Changes</h3>

<ol>
  <li><strong>Variable type</strong>: Changed from <code class="language-plaintext highlighter-rouge">int</code> to <code class="language-plaintext highlighter-rouge">int64</code> (required for atomic operations)</li>
  <li><strong>Increment operation</strong>: <code class="language-plaintext highlighter-rouge">updatedCount++</code> → <code class="language-plaintext highlighter-rouge">atomic.AddInt64(&amp;updatedCount, 1)</code></li>
  <li><strong>Reading the value</strong>: <code class="language-plaintext highlighter-rouge">updatedCount</code> → <code class="language-plaintext highlighter-rouge">atomic.LoadInt64(&amp;updatedCount)</code></li>
</ol>

<h2 id="how-atomic-operations-work">How Atomic Operations Work</h2>

<h3 id="atomicaddint64updatedcount-1"><code class="language-plaintext highlighter-rouge">atomic.AddInt64(&amp;updatedCount, 1)</code></h3>

<p>This function provides several guarantees:</p>

<ul>
  <li><strong>Atomically adds 1</strong> to the value at the specified memory address</li>
  <li><strong>Cannot be interrupted</strong> by other goroutines or CPU context switches</li>
  <li><strong>Hardware-level guarantee</strong> that the entire operation completes as a single unit</li>
  <li><strong>Returns the new value</strong> after the addition (though we don’t use it in our example)</li>
  <li><strong>Thread-safe</strong> across all CPU cores</li>
</ul>

<h3 id="atomicloadint64updatedcount"><code class="language-plaintext highlighter-rouge">atomic.LoadInt64(&amp;updatedCount)</code></h3>

<p>This function ensures safe reading:</p>

<ul>
  <li><strong>Atomically reads</strong> the complete value from memory</li>
  <li><strong>Guarantees consistency</strong> - you’ll never see a partially updated value</li>
  <li><strong>Prevents compiler optimizations</strong> that might cache stale values</li>
  <li><strong>Provides memory synchronization</strong> with atomic writes</li>
</ul>

<h2 id="why-atomic-operations-prevent-race-conditions">Why Atomic Operations Prevent Race Conditions</h2>

<p>The magic of atomic operations lies in their <strong>indivisible nature</strong>:</p>

<ol>
  <li><strong>Single CPU instruction</strong>: <code class="language-plaintext highlighter-rouge">atomic.AddInt64</code> translates to a single CPU instruction that cannot be interrupted</li>
  <li><strong>Memory barriers</strong>: Atomic operations include memory synchronization, ensuring all CPU cores see consistent values</li>
  <li><strong>No lost updates</strong>: Multiple goroutines can safely increment the same counter simultaneously</li>
  <li><strong>Ordering guarantees</strong>: Atomic operations provide happens-before relationships for memory ordering</li>
</ol>

<h2 id="benchmarking-the-performance">Benchmarking the Performance</h2>

<p>I was curious about how fast these atomic operations really are, so I wrote a benchmark. The results blew my mind:</p>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span>

<span class="k">import</span> <span class="p">(</span>
    <span class="s">"sync"</span>
    <span class="s">"sync/atomic"</span>
    <span class="s">"testing"</span>
<span class="p">)</span>

<span class="k">func</span> <span class="n">BenchmarkAtomicCounter</span><span class="p">(</span><span class="n">b</span> <span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">B</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">var</span> <span class="n">counter</span> <span class="kt">int64</span>
    <span class="k">var</span> <span class="n">wg</span> <span class="n">sync</span><span class="o">.</span><span class="n">WaitGroup</span>
    
    <span class="n">numGoroutines</span> <span class="o">:=</span> <span class="m">100</span>
    <span class="n">incrementsPerGoroutine</span> <span class="o">:=</span> <span class="n">b</span><span class="o">.</span><span class="n">N</span> <span class="o">/</span> <span class="n">numGoroutines</span>
    
    <span class="k">for</span> <span class="n">i</span> <span class="o">:=</span> <span class="m">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">numGoroutines</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span> <span class="p">{</span>
        <span class="n">wg</span><span class="o">.</span><span class="n">Add</span><span class="p">(</span><span class="m">1</span><span class="p">)</span>
        <span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
            <span class="k">defer</span> <span class="n">wg</span><span class="o">.</span><span class="n">Done</span><span class="p">()</span>
            <span class="k">for</span> <span class="n">j</span> <span class="o">:=</span> <span class="m">0</span><span class="p">;</span> <span class="n">j</span> <span class="o">&lt;</span> <span class="n">incrementsPerGoroutine</span><span class="p">;</span> <span class="n">j</span><span class="o">++</span> <span class="p">{</span>
                <span class="n">atomic</span><span class="o">.</span><span class="n">AddInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="n">counter</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span>
            <span class="p">}</span>
        <span class="p">}()</span>
    <span class="p">}</span>
    
    <span class="n">wg</span><span class="o">.</span><span class="n">Wait</span><span class="p">()</span>
    <span class="n">expected</span> <span class="o">:=</span> <span class="kt">int64</span><span class="p">(</span><span class="n">numGoroutines</span> <span class="o">*</span> <span class="n">incrementsPerGoroutine</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">atomic</span><span class="o">.</span><span class="n">LoadInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="n">counter</span><span class="p">)</span> <span class="o">!=</span> <span class="n">expected</span> <span class="p">{</span>
        <span class="n">b</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"Counter mismatch: got %d, want %d"</span><span class="p">,</span> 
                 <span class="n">atomic</span><span class="o">.</span><span class="n">LoadInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="n">counter</span><span class="p">),</span> <span class="n">expected</span><span class="p">)</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>On my machine, this benchmark shows atomic operations can handle millions of concurrent increments per second with perfect accuracy. That’s faster than I ever expected!</p>

<p>Here are the actual results on my Apple M3 Pro:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>goos: darwin
goarch: arm64
pkg: test
cpu: Apple M3 Pro
BenchmarkAtomicCounter-12    	47625759	        27.27 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	test	2.313s
</code></pre></div></div>

<p><strong>That’s 47.6 million operations in just over 2 seconds!</strong> Each atomic increment takes only 27 nanoseconds with zero memory allocations. To put this in perspective, that’s fast enough to handle the busiest web applications without breaking a sweat.</p>

<h2 id="alternative-solutions-compared">Alternative Solutions Compared</h2>

<p>While atomic operations are perfect for our use case, it’s worth understanding the alternatives:</p>

<h3 id="1-mutex-more-general-heavier">1. <strong>Mutex (More General, Heavier)</strong></h3>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">var</span> <span class="n">mu</span> <span class="n">sync</span><span class="o">.</span><span class="n">Mutex</span>
<span class="k">var</span> <span class="n">updatedCount</span> <span class="kt">int</span>

<span class="c">// In each goroutine:</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="no">nil</span> <span class="p">{</span>
    <span class="n">mu</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span>
    <span class="n">updatedCount</span><span class="o">++</span>
    <span class="n">mu</span><span class="o">.</span><span class="n">Unlock</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>

<p><strong>Pros:</strong></p>
<ul>
  <li>Works with any data type</li>
  <li>Can protect complex operations</li>
  <li>Familiar to developers from other languages</li>
</ul>

<p><strong>Cons:</strong></p>
<ul>
  <li>Higher overhead (OS-level synchronization)</li>
  <li>Risk of deadlocks with multiple mutexes</li>
  <li>Goroutines block waiting for the lock</li>
</ul>

<h3 id="2-channel-the-go-way">2. <strong>Channel (The “Go Way”)</strong></h3>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">updateChan</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="k">struct</span><span class="p">{},</span> <span class="nb">len</span><span class="p">(</span><span class="n">instances</span><span class="p">))</span>

<span class="c">// In each goroutine:</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">==</span> <span class="no">nil</span> <span class="p">{</span>
    <span class="n">updateChan</span> <span class="o">&lt;-</span> <span class="k">struct</span><span class="p">{}{}</span>
<span class="p">}</span>

<span class="c">// After all goroutines complete:</span>
<span class="nb">close</span><span class="p">(</span><span class="n">updateChan</span><span class="p">)</span>
<span class="n">updatedCount</span> <span class="o">:=</span> <span class="m">0</span>
<span class="k">for</span> <span class="k">range</span> <span class="n">updateChan</span> <span class="p">{</span>
    <span class="n">updatedCount</span><span class="o">++</span>
<span class="p">}</span>
</code></pre></div></div>

<p><strong>Pros:</strong></p>
<ul>
  <li>Idiomatic Go</li>
  <li>Natural backpressure with buffered channels</li>
  <li>Composable with <code class="language-plaintext highlighter-rouge">select</code> statements</li>
</ul>

<p><strong>Cons:</strong></p>
<ul>
  <li>More complex for simple counting</li>
  <li>Memory overhead for the channel buffer</li>
  <li>Requires careful channel management</li>
</ul>

<h3 id="3-return-values-and-sum">3. <strong>Return Values and Sum</strong></h3>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="n">result</span> <span class="k">struct</span> <span class="p">{</span>
    <span class="n">success</span> <span class="kt">bool</span>
    <span class="n">err</span>     <span class="kt">error</span>
<span class="p">}</span>

<span class="n">results</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">([]</span><span class="n">result</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">instances</span><span class="p">))</span>

<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">instance</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">instances</span> <span class="p">{</span>
    <span class="n">i</span><span class="p">,</span> <span class="n">instance</span> <span class="o">:=</span> <span class="n">i</span><span class="p">,</span> <span class="n">instance</span>
    <span class="n">wg</span><span class="o">.</span><span class="n">Go</span><span class="p">(</span><span class="k">func</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
        <span class="n">err</span> <span class="o">:=</span> <span class="n">updateInstance</span><span class="p">(</span><span class="n">instance</span><span class="p">)</span>
        <span class="n">results</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">result</span><span class="p">{</span><span class="n">success</span><span class="o">:</span> <span class="n">err</span> <span class="o">==</span> <span class="no">nil</span><span class="p">,</span> <span class="n">err</span><span class="o">:</span> <span class="n">err</span><span class="p">}</span>
        <span class="k">return</span> <span class="n">err</span>
    <span class="p">})</span>
<span class="p">}</span>

<span class="c">// Count successes</span>
<span class="n">updatedCount</span> <span class="o">:=</span> <span class="m">0</span>
<span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">r</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">results</span> <span class="p">{</span>
    <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">success</span> <span class="p">{</span>
        <span class="n">updatedCount</span><span class="o">++</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p><strong>Pros:</strong></p>
<ul>
  <li>No shared state, no synchronization needed</li>
  <li>Easy to collect additional information</li>
  <li>Functional programming style</li>
</ul>

<p><strong>Cons:</strong></p>
<ul>
  <li>Requires pre-allocated slice</li>
  <li>Memory usage grows with number of operations</li>
  <li>Less efficient for simple counting</li>
</ul>

<h2 id="when-to-choose-atomic-operations">When to Choose Atomic Operations</h2>

<p>Choose <code class="language-plaintext highlighter-rouge">sync/atomic</code> when:</p>

<ul>
  <li>✅ <strong>Simple numeric operations</strong>: Counters, flags, sums</li>
  <li>✅ <strong>High-performance requirements</strong>: Minimal overhead</li>
  <li>✅ <strong>Lock-free algorithms</strong>: No blocking behavior needed</li>
  <li>✅ <strong>Frequent updates</strong>: Many goroutines updating the same value</li>
</ul>

<p>Avoid <code class="language-plaintext highlighter-rouge">sync/atomic</code> when:</p>

<ul>
  <li>❌ <strong>Complex data structures</strong>: Use mutexes instead</li>
  <li>❌ <strong>Multiple related operations</strong>: Atomic operations are for single values</li>
  <li>❌ <strong>Need transaction semantics</strong>: Use mutexes or channels for multi-step operations</li>
</ul>

<h2 id="best-practices-for-atomic-operations">Best Practices for Atomic Operations</h2>

<ol>
  <li><strong>Use the correct integer type</strong>: <code class="language-plaintext highlighter-rouge">int32</code>, <code class="language-plaintext highlighter-rouge">int64</code>, <code class="language-plaintext highlighter-rouge">uint32</code>, <code class="language-plaintext highlighter-rouge">uint64</code>, <code class="language-plaintext highlighter-rouge">uintptr</code></li>
  <li><strong>Be consistent</strong>: Always use atomic operations for a variable, never mix with regular operations</li>
  <li><strong>Align memory</strong>: Use <code class="language-plaintext highlighter-rouge">atomic.Value</code> for complex types</li>
  <li><strong>Document clearly</strong>: Make it obvious that a variable requires atomic access</li>
</ol>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Good: Clear documentation and consistent usage</span>
<span class="k">type</span> <span class="n">ServiceMetrics</span> <span class="k">struct</span> <span class="p">{</span>
    <span class="c">// requestCount must be accessed using atomic operations only</span>
    <span class="n">requestCount</span> <span class="kt">int64</span>
    
    <span class="c">// errorCount must be accessed using atomic operations only  </span>
    <span class="n">errorCount</span>   <span class="kt">int64</span>
<span class="p">}</span>

<span class="k">func</span> <span class="p">(</span><span class="n">m</span> <span class="o">*</span><span class="n">ServiceMetrics</span><span class="p">)</span> <span class="n">IncrementRequests</span><span class="p">()</span> <span class="p">{</span>
    <span class="n">atomic</span><span class="o">.</span><span class="n">AddInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="n">m</span><span class="o">.</span><span class="n">requestCount</span><span class="p">,</span> <span class="m">1</span><span class="p">)</span>
<span class="p">}</span>

<span class="k">func</span> <span class="p">(</span><span class="n">m</span> <span class="o">*</span><span class="n">ServiceMetrics</span><span class="p">)</span> <span class="n">GetRequestCount</span><span class="p">()</span> <span class="kt">int64</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">atomic</span><span class="o">.</span><span class="n">LoadInt64</span><span class="p">(</span><span class="o">&amp;</span><span class="n">m</span><span class="o">.</span><span class="n">requestCount</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>

<h2 id="conclusion">Conclusion</h2>

<p>Race conditions are one of the most subtle and dangerous bugs in concurrent programming, but Go’s <code class="language-plaintext highlighter-rouge">sync/atomic</code> package provides an elegant solution for simple cases like counters and flags.</p>

<p><strong>Key takeaways:</strong></p>

<ul>
  <li><strong>Race conditions occur</strong> when multiple goroutines access shared data without proper synchronization</li>
  <li><strong>Atomic operations</strong> provide lock-free, thread-safe access to simple data types</li>
  <li><strong>Choose the right tool</strong>: Atomic for simple operations, mutexes for complex logic, channels for communication</li>
  <li><strong>Performance matters</strong>: Atomic operations are typically the fastest synchronization mechanism</li>
  <li><strong>Always be consistent</strong>: Once you choose atomic operations for a variable, use them everywhere</li>
</ul>

<p>The next time you’re building concurrent Go applications, remember that a simple counter can hide complex race conditions. But with atomic operations, you can count on accurate results! 🏁</p>

<h2 id="further-reading">Further Reading</h2>

<ul>
  <li><a href="https://pkg.go.dev/sync/atomic">Go atomic package documentation</a></li>
  <li><a href="https://golang.org/ref/mem">The Go Memory Model</a></li>
  <li><a href="https://golang.org/doc/effective_go#concurrency">Effective Go - Concurrency</a></li>
</ul>

<hr />

<p><em>What’s your experience with race conditions in Go? Have you encountered subtle bugs that atomic operations helped solve? Share your stories in the comments below!</em></p>]]></content><author><name>Arshad Mehmood</name></author><category term="Development" /><category term="go" /><category term="concurrency" /><category term="race-conditions" /><category term="atomic" /><category term="goroutines" /><summary type="html"><![CDATA[Learn how to use Go's sync/atomic package to eliminate race conditions when working with concurrent goroutines. A practical guide with real-world examples.]]></summary></entry><entry><title type="html">Obsidian Notes Linker Open Source</title><link href="https://arshadmehmood.com/obsidian-notes-linker-open-source/" rel="alternate" type="text/html" title="Obsidian Notes Linker Open Source" /><published>2025-01-28T00:00:00+00:00</published><updated>2025-01-28T00:00:00+00:00</updated><id>https://arshadmehmood.com/obsidian-notes-linker-open-source</id><content type="html" xml:base="https://arshadmehmood.com/obsidian-notes-linker-open-source/"><![CDATA[<hr />
<p>title: “Obsidian Notes Linker - Open Source Tool Released”
category: Development
tags:</p>
<ul>
  <li>obsidian</li>
  <li>notes</li>
  <li>productivity</li>
  <li>open-source</li>
  <li>typescript</li>
  <li>electron
header:
  image: /assets/images/posts/obsidian-notes-linker.png
  teaser: /assets/images/posts/obsidian-notes-linker.png
comments: true
toc_sticky: true
excerpt: “Introducing Obsidian Notes Linker - an open-source tool to automatically create bidirectional links between your Obsidian notes. Boost your knowledge management workflow.”
—Obsidian Notes Linker is a new open-source project designed to help users of Obsidian, a popular markdown-based note-taking app, to automatically link their markdown files. This tool is written in Python and is available on GitHub for anyone to use and contribute to.</li>
</ul>

<p>Project can be found on Github <a href="https://github.com/arshad115/obsidian-linker">Obsidian Linker</a></p>

<h2 id="features">Features</h2>

<ul>
  <li><strong>Automatic Linking</strong>: The tool scans your markdown files and automatically creates links between notes based on their content.</li>
  <li><strong>Efficient</strong>: Handles large collections of notes quickly and efficiently.</li>
  <li><strong>Uses Wikilink Format</strong>: Links are created using the Wikilink format, which is compatible with Obsidian.</li>
  <li><strong>Skip Links in Metadata</strong>: The tool intelligently skips linking within metadata sections of your notes.</li>
  <li><strong>Skip Links in Codeblocks and Inline Code</strong>: Ensures that links are not created within code blocks or inline code, preserving the integrity of your code snippets.</li>
</ul>

<h2 id="installation">Installation</h2>

<p>To install the Obsidian Notes Linker, you can clone the repository from GitHub and install the required dependencies:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/arshad115/obsidian-linker.git
<span class="nb">cd </span>obsidian-linker
pip <span class="nb">install</span> <span class="nt">-r</span> requirements.txt
</code></pre></div></div>

<h2 id="usage">Usage</h2>

<p>To use the tool, simply run the following command in your terminal:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python obsidianlinker.py /path/to/vault/
</code></pre></div></div>
<p><img src="https://arshadmehmood.com/assets/images/posts/ol-usage.png" alt="Usage" /></p>

<p>This will scan your notes and add links where appropriate.</p>

<h2 id="example">Example</h2>

<p>I ran the obsidian linker on my digital garden with 430 markdown notes and it added 17,191 links. I had not linked notes except for the ones in one folder and it did an amazing job of making the graph travel easy and it just looks beautiful!</p>

<h3 id="before-linking">Before Linking</h3>
<p><img src="https://arshadmehmood.com/assets/images/posts/ol-before.png" alt="Before Linking" /></p>

<h3 id="after-linking">After Linking</h3>
<p><img src="https://arshadmehmood.com/assets/images/posts/ol-after.png" alt="After Linking" /></p>

<blockquote>
  <p><strong>Warning</strong>: Make sure to back up your vault before using this tool, as it can make irreversible edits.</p>
</blockquote>

<h2 id="contributing">Contributing</h2>

<p>Contributions are welcome! If you have any ideas for new features or improvements, feel free to open an issue or submit a pull request on <a href="https://github.com/arshad115/obsidian-linker">Obsidian Linker GitHub</a>.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Obsidian Notes Linker is a powerful tool for anyone looking to enhance their note-taking experience in Obsidian. By automating the process of linking notes, it saves time and helps you create a more interconnected and useful knowledge base. Check out the project on GitHub and start linking your notes today!</p>]]></content><author><name>Arshad Mehmood</name></author><summary type="html"><![CDATA[title: “Obsidian Notes Linker - Open Source Tool Released” category: Development tags: obsidian notes productivity open-source typescript electron header: image: /assets/images/posts/obsidian-notes-linker.png teaser: /assets/images/posts/obsidian-notes-linker.png comments: true toc_sticky: true excerpt: “Introducing Obsidian Notes Linker - an open-source tool to automatically create bidirectional links between your Obsidian notes. Boost your knowledge management workflow.” —Obsidian Notes Linker is a new open-source project designed to help users of Obsidian, a popular markdown-based note-taking app, to automatically link their markdown files. This tool is written in Python and is available on GitHub for anyone to use and contribute to.]]></summary></entry><entry><title type="html">Setup Prometheus Exporter in Jenkins</title><link href="https://arshadmehmood.com/devops/setup-prometheus-exporter-jenkins/" rel="alternate" type="text/html" title="Setup Prometheus Exporter in Jenkins" /><published>2024-10-11T00:00:00+00:00</published><updated>2024-10-11T00:00:00+00:00</updated><id>https://arshadmehmood.com/devops/setup-prometheus-exporter-jenkins</id><content type="html" xml:base="https://arshadmehmood.com/devops/setup-prometheus-exporter-jenkins/"><![CDATA[<p><em><strong>Note</strong> This post is part of the <a href="/software/devops-journey/">DevOps Journey</a></em></p>

<p>To monitor Jenkins with Prometheus, we need to expose Jenkins metrics. Jenkins has a built-in Prometheus metrics endpoint that we can enable.</p>

<h3 id="install-prometheus-metrics-plugin">Install Prometheus Metrics Plugin</h3>

<ol>
  <li>Go to Jenkins Dashboard → Manage Jenkins → Manage Plugins</li>
  <li>Search for “Prometheus metrics” plugin</li>
  <li>Install and restart Jenkins</li>
</ol>

<p><img src="https://arshadmehmood.com/assets/images/posts/jenkins-prometheus.png" alt="Jenkins Prometheus Plugin" /></p>

<h3 id="configure-prometheus-metrics">Configure Prometheus Metrics</h3>

<p>After installation, Jenkins will automatically expose metrics at:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://your-jenkins-url/prometheus/
</code></pre></div></div>

<h3 id="access-metrics-endpoint">Access Metrics Endpoint</h3>

<p>You can verify the metrics are being exposed by visiting:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://localhost:8080/prometheus/
</code></pre></div></div>

<p>You should see output like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># HELP jenkins_builds_duration_milliseconds_summary Build times in milliseconds
# TYPE jenkins_builds_duration_milliseconds_summary summary
jenkins_builds_duration_milliseconds_summary{quantile="0.5",} 1234.0
jenkins_builds_duration_milliseconds_summary{quantile="0.95",} 5678.0
</code></pre></div></div>

<h3 id="configure-prometheus-to-scrape-jenkins">Configure Prometheus to Scrape Jenkins</h3>

<p>Add Jenkins as a target in your <code class="language-plaintext highlighter-rouge">prometheus.yml</code>:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">global</span><span class="pi">:</span>
  <span class="na">scrape_interval</span><span class="pi">:</span> <span class="s">15s</span>

<span class="na">scrape_configs</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">job_name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">jenkins'</span>
    <span class="na">static_configs</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">targets</span><span class="pi">:</span> <span class="pi">[</span><span class="s1">'</span><span class="s">localhost:8080'</span><span class="pi">]</span>
    <span class="na">metrics_path</span><span class="pi">:</span> <span class="s1">'</span><span class="s">/prometheus'</span>
    <span class="na">scrape_interval</span><span class="pi">:</span> <span class="s">5s</span>
</code></pre></div></div>

<h3 id="key-jenkins-metrics">Key Jenkins Metrics</h3>

<p>The plugin exposes various useful metrics:</p>

<ul>
  <li><strong>Build Metrics</strong>: <code class="language-plaintext highlighter-rouge">jenkins_builds_duration_milliseconds_summary</code></li>
  <li><strong>Job Metrics</strong>: <code class="language-plaintext highlighter-rouge">jenkins_job_count_total</code></li>
  <li><strong>Queue Metrics</strong>: <code class="language-plaintext highlighter-rouge">jenkins_queue_size_value</code></li>
  <li><strong>Node Metrics</strong>: <code class="language-plaintext highlighter-rouge">jenkins_node_count_value</code></li>
  <li><strong>Executor Metrics</strong>: <code class="language-plaintext highlighter-rouge">jenkins_executor_count_value</code></li>
</ul>

<h3 id="restart-prometheus">Restart Prometheus</h3>

<p>After updating the configuration:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker restart prometheus
</code></pre></div></div>

<h3 id="verify-in-prometheus-ui">Verify in Prometheus UI</h3>

<ol>
  <li>Open Prometheus UI at http://localhost:9090</li>
  <li>Go to Status → Targets</li>
  <li>Verify Jenkins target is “UP”</li>
  <li>Query some metrics like <code class="language-plaintext highlighter-rouge">jenkins_builds_duration_milliseconds_summary</code></li>
</ol>

<p><img src="https://arshadmehmood.com/assets/images/posts/prometheus-targets.png" alt="Prometheus Targets" /></p>

<h3 id="grafana-dashboard">Grafana Dashboard</h3>

<p>Import Jenkins dashboard ID <code class="language-plaintext highlighter-rouge">9964</code> from Grafana’s dashboard repository for pre-built Jenkins visualizations.</p>]]></content><author><name>Arshad Mehmood</name></author><category term="DevOps" /><category term="devops" /><category term="prometheus" /><category term="jenkins" /><category term="monitoring" /><category term="metrics" /><summary type="html"><![CDATA[Note This post is part of the DevOps Journey]]></summary></entry><entry><title type="html">Setup SonarQube Prometheus Exporter</title><link href="https://arshadmehmood.com/devops/setup-sonarqube-exporter-jenkins/" rel="alternate" type="text/html" title="Setup SonarQube Prometheus Exporter" /><published>2024-10-11T00:00:00+00:00</published><updated>2024-10-11T00:00:00+00:00</updated><id>https://arshadmehmood.com/devops/setup-sonarqube-exporter-jenkins</id><content type="html" xml:base="https://arshadmehmood.com/devops/setup-sonarqube-exporter-jenkins/"><![CDATA[<p><em><strong>Note</strong> This post is part of the <a href="/software/devops-journey/">DevOps Journey</a></em></p>

<p>To monitor SonarQube metrics with Prometheus, we can use the SonarQube Prometheus Exporter plugin or the built-in web API metrics endpoint.</p>

<h3 id="method-1-sonarqube-prometheus-exporter-plugin">Method 1: SonarQube Prometheus Exporter Plugin</h3>

<h4 id="install-the-plugin">Install the Plugin</h4>

<ol>
  <li>Download the SonarQube Prometheus Exporter plugin from GitHub</li>
  <li>Place the JAR file in <code class="language-plaintext highlighter-rouge">$SONARQUBE_HOME/extensions/plugins/</code></li>
  <li>Restart SonarQube</li>
</ol>

<h4 id="access-metrics">Access Metrics</h4>

<p>The plugin exposes metrics at:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://localhost:9000/api/prometheus/metrics
</code></pre></div></div>

<h3 id="method-2-using-sonarqube-web-api">Method 2: Using SonarQube Web API</h3>

<p>SonarQube provides built-in APIs that can be scraped by Prometheus using custom exporters.</p>

<h4 id="key-sonarqube-metrics-apis">Key SonarQube Metrics APIs</h4>

<ul>
  <li><strong>System Health</strong>: <code class="language-plaintext highlighter-rouge">/api/system/health</code></li>
  <li><strong>Project Statistics</strong>: <code class="language-plaintext highlighter-rouge">/api/measures/component</code></li>
  <li><strong>Quality Gates</strong>: <code class="language-plaintext highlighter-rouge">/api/qualitygates/project_status</code></li>
  <li><strong>Issues</strong>: <code class="language-plaintext highlighter-rouge">/api/issues/search</code></li>
</ul>

<h3 id="configure-prometheus-to-scrape-sonarqube">Configure Prometheus to Scrape SonarQube</h3>

<p>Add SonarQube to your <code class="language-plaintext highlighter-rouge">prometheus.yml</code>:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">scrape_configs</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">job_name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">sonarqube'</span>
    <span class="na">static_configs</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">targets</span><span class="pi">:</span> <span class="pi">[</span><span class="s1">'</span><span class="s">localhost:9000'</span><span class="pi">]</span>
    <span class="na">metrics_path</span><span class="pi">:</span> <span class="s1">'</span><span class="s">/api/prometheus/metrics'</span>
    <span class="na">scrape_interval</span><span class="pi">:</span> <span class="s">30s</span>
    <span class="na">basic_auth</span><span class="pi">:</span>
      <span class="na">username</span><span class="pi">:</span> <span class="s1">'</span><span class="s">your-sonarqube-token'</span>
      <span class="na">password</span><span class="pi">:</span> <span class="s1">'</span><span class="s">'</span>
</code></pre></div></div>

<p><img src="https://arshadmehmood.com/assets/images/posts/prometheus.png" alt="Prometheus Configuration" /></p>

<h3 id="custom-exporter-script">Custom Exporter Script</h3>

<p>If the plugin isn’t available, create a custom Python exporter:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#!/usr/bin/env python3
</span><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">from</span> <span class="nn">prometheus_client</span> <span class="kn">import</span> <span class="n">start_http_server</span><span class="p">,</span> <span class="n">Gauge</span>

<span class="c1"># SonarQube connection
</span><span class="n">SONARQUBE_URL</span> <span class="o">=</span> <span class="s">"http://localhost:9000"</span>
<span class="n">TOKEN</span> <span class="o">=</span> <span class="s">"your-sonarqube-token"</span>

<span class="c1"># Prometheus metrics
</span><span class="n">lines_of_code</span> <span class="o">=</span> <span class="n">Gauge</span><span class="p">(</span><span class="s">'sonarqube_lines_of_code'</span><span class="p">,</span> <span class="s">'Lines of code'</span><span class="p">,</span> <span class="p">[</span><span class="s">'project'</span><span class="p">])</span>
<span class="n">bugs</span> <span class="o">=</span> <span class="n">Gauge</span><span class="p">(</span><span class="s">'sonarqube_bugs'</span><span class="p">,</span> <span class="s">'Number of bugs'</span><span class="p">,</span> <span class="p">[</span><span class="s">'project'</span><span class="p">])</span>
<span class="n">vulnerabilities</span> <span class="o">=</span> <span class="n">Gauge</span><span class="p">(</span><span class="s">'sonarqube_vulnerabilities'</span><span class="p">,</span> <span class="s">'Number of vulnerabilities'</span><span class="p">,</span> <span class="p">[</span><span class="s">'project'</span><span class="p">])</span>
<span class="n">code_smells</span> <span class="o">=</span> <span class="n">Gauge</span><span class="p">(</span><span class="s">'sonarqube_code_smells'</span><span class="p">,</span> <span class="s">'Number of code smells'</span><span class="p">,</span> <span class="p">[</span><span class="s">'project'</span><span class="p">])</span>

<span class="k">def</span> <span class="nf">collect_metrics</span><span class="p">():</span>
    <span class="c1"># Get projects
</span>    <span class="n">projects_url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">SONARQUBE_URL</span><span class="si">}</span><span class="s">/api/projects/search"</span>
    <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">projects_url</span><span class="p">,</span> <span class="n">auth</span><span class="o">=</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">,</span> <span class="s">''</span><span class="p">))</span>
    
    <span class="k">for</span> <span class="n">project</span> <span class="ow">in</span> <span class="n">response</span><span class="p">.</span><span class="n">json</span><span class="p">()[</span><span class="s">'components'</span><span class="p">]:</span>
        <span class="n">project_key</span> <span class="o">=</span> <span class="n">project</span><span class="p">[</span><span class="s">'key'</span><span class="p">]</span>
        
        <span class="c1"># Get metrics for each project
</span>        <span class="n">metrics_url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">SONARQUBE_URL</span><span class="si">}</span><span class="s">/api/measures/component"</span>
        <span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
            <span class="s">'component'</span><span class="p">:</span> <span class="n">project_key</span><span class="p">,</span>
            <span class="s">'metricKeys'</span><span class="p">:</span> <span class="s">'ncloc,bugs,vulnerabilities,code_smells'</span>
        <span class="p">}</span>
        
        <span class="n">metrics_response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">metrics_url</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">params</span><span class="p">,</span> <span class="n">auth</span><span class="o">=</span><span class="p">(</span><span class="n">TOKEN</span><span class="p">,</span> <span class="s">''</span><span class="p">))</span>
        <span class="n">measures</span> <span class="o">=</span> <span class="n">metrics_response</span><span class="p">.</span><span class="n">json</span><span class="p">()[</span><span class="s">'component'</span><span class="p">][</span><span class="s">'measures'</span><span class="p">]</span>
        
        <span class="k">for</span> <span class="n">measure</span> <span class="ow">in</span> <span class="n">measures</span><span class="p">:</span>
            <span class="n">metric_key</span> <span class="o">=</span> <span class="n">measure</span><span class="p">[</span><span class="s">'metric'</span><span class="p">]</span>
            <span class="n">value</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">measure</span><span class="p">[</span><span class="s">'value'</span><span class="p">])</span>
            
            <span class="k">if</span> <span class="n">metric_key</span> <span class="o">==</span> <span class="s">'ncloc'</span><span class="p">:</span>
                <span class="n">lines_of_code</span><span class="p">.</span><span class="n">labels</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="n">project_key</span><span class="p">).</span><span class="nb">set</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
            <span class="k">elif</span> <span class="n">metric_key</span> <span class="o">==</span> <span class="s">'bugs'</span><span class="p">:</span>
                <span class="n">bugs</span><span class="p">.</span><span class="n">labels</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="n">project_key</span><span class="p">).</span><span class="nb">set</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
            <span class="k">elif</span> <span class="n">metric_key</span> <span class="o">==</span> <span class="s">'vulnerabilities'</span><span class="p">:</span>
                <span class="n">vulnerabilities</span><span class="p">.</span><span class="n">labels</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="n">project_key</span><span class="p">).</span><span class="nb">set</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
            <span class="k">elif</span> <span class="n">metric_key</span> <span class="o">==</span> <span class="s">'code_smells'</span><span class="p">:</span>
                <span class="n">code_smells</span><span class="p">.</span><span class="n">labels</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="n">project_key</span><span class="p">).</span><span class="nb">set</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>

<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
    <span class="n">start_http_server</span><span class="p">(</span><span class="mi">8000</span><span class="p">)</span>
    <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
        <span class="n">collect_metrics</span><span class="p">()</span>
        <span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">60</span><span class="p">)</span>
</code></pre></div></div>

<h3 id="run-the-custom-exporter">Run the Custom Exporter</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python3 sonarqube_exporter.py
</code></pre></div></div>

<h3 id="update-prometheus-configuration">Update Prometheus Configuration</h3>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">scrape_configs</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">job_name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">sonarqube-custom'</span>
    <span class="na">static_configs</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">targets</span><span class="pi">:</span> <span class="pi">[</span><span class="s1">'</span><span class="s">localhost:8000'</span><span class="pi">]</span>
    <span class="na">scrape_interval</span><span class="pi">:</span> <span class="s">60s</span>
</code></pre></div></div>

<h3 id="key-sonarqube-metrics">Key SonarQube Metrics</h3>

<p>Monitor these important metrics:</p>

<ul>
  <li><strong>Lines of Code</strong> (<code class="language-plaintext highlighter-rouge">ncloc</code>)</li>
  <li><strong>Bugs</strong> (<code class="language-plaintext highlighter-rouge">bugs</code>)</li>
  <li><strong>Vulnerabilities</strong> (<code class="language-plaintext highlighter-rouge">vulnerabilities</code>)</li>
  <li><strong>Code Smells</strong> (<code class="language-plaintext highlighter-rouge">code_smells</code>)</li>
  <li><strong>Coverage</strong> (<code class="language-plaintext highlighter-rouge">coverage</code>)</li>
  <li><strong>Duplicated Lines</strong> (<code class="language-plaintext highlighter-rouge">duplicated_lines_density</code>)</li>
  <li><strong>Technical Debt</strong> (<code class="language-plaintext highlighter-rouge">sqale_index</code>)</li>
</ul>

<h3 id="grafana-dashboard">Grafana Dashboard</h3>

<p>Create dashboards to visualize:</p>

<ul>
  <li>Code quality trends over time</li>
  <li>Project comparison matrices</li>
  <li>Quality gate success rates</li>
  <li>Technical debt accumulation</li>
</ul>

<p>This setup provides comprehensive monitoring of your code quality metrics alongside your infrastructure monitoring.</p>]]></content><author><name>Arshad Mehmood</name></author><category term="DevOps" /><category term="devops" /><category term="sonarqube" /><category term="prometheus" /><category term="monitoring" /><category term="code-quality" /><summary type="html"><![CDATA[Note This post is part of the DevOps Journey]]></summary></entry><entry><title type="html">Setup SonarQube with Jenkins</title><link href="https://arshadmehmood.com/devops/setup-sonarqube-jenkins/" rel="alternate" type="text/html" title="Setup SonarQube with Jenkins" /><published>2024-10-11T00:00:00+00:00</published><updated>2024-10-11T00:00:00+00:00</updated><id>https://arshadmehmood.com/devops/setup-sonarqube-jenkins</id><content type="html" xml:base="https://arshadmehmood.com/devops/setup-sonarqube-jenkins/"><![CDATA[<p><em><strong>Note</strong> This post is part of the <a href="/software/devops-journey/">DevOps Journey</a></em></p>

<p>SonarQube provides continuous inspection of code quality through static analysis. Let’s integrate it with Jenkins for automated code quality checks.</p>

<h3 id="run-sonarqube-with-docker">Run SonarQube with Docker</h3>

<p>Start SonarQube using Docker:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">-d</span> <span class="nt">--name</span> sonarqube <span class="se">\</span>
  <span class="nt">-p</span> 9000:9000 <span class="se">\</span>
  <span class="nt">-e</span> <span class="nv">SONAR_ES_BOOTSTRAP_CHECKS_DISABLE</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
  sonarqube:latest
</code></pre></div></div>

<p>Wait for SonarQube to start (check logs with <code class="language-plaintext highlighter-rouge">docker logs sonarqube</code>), then access it at http://localhost:9000</p>

<p><strong>Default credentials:</strong> admin/admin (change on first login)</p>

<h3 id="install-sonarqube-scanner-plugin-in-jenkins">Install SonarQube Scanner Plugin in Jenkins</h3>

<ol>
  <li>Go to Jenkins Dashboard → Manage Jenkins → Manage Plugins</li>
  <li>Search for “SonarQube Scanner” plugin</li>
  <li>Install and restart Jenkins</li>
</ol>

<h3 id="configure-sonarqube-server-in-jenkins">Configure SonarQube Server in Jenkins</h3>

<ol>
  <li>Navigate to Manage Jenkins → Configure System</li>
  <li>Scroll to “SonarQube servers” section</li>
  <li>Click “Add SonarQube”</li>
  <li>Configure:
    <ul>
      <li>Name: <code class="language-plaintext highlighter-rouge">SonarQube</code></li>
      <li>Server URL: <code class="language-plaintext highlighter-rouge">http://localhost:9000</code></li>
      <li>Server authentication token: (generate in SonarQube)</li>
    </ul>
  </li>
</ol>

<h3 id="generate-sonarqube-token">Generate SonarQube Token</h3>

<ol>
  <li>Login to SonarQube (http://localhost:9000)</li>
  <li>Go to User → My Account → Security</li>
  <li>Generate a new token</li>
  <li>Copy the token for Jenkins configuration</li>
</ol>

<p><img src="https://arshadmehmood.com/assets/images/posts/sonarqube-token.png" alt="SonarQube Token Generation" /></p>

<h3 id="add-token-to-jenkins">Add Token to Jenkins</h3>

<ol>
  <li>In Jenkins SonarQube configuration, click “Add” next to Server authentication token</li>
  <li>Select “Secret text”</li>
  <li>Paste the SonarQube token</li>
  <li>Give it an ID like <code class="language-plaintext highlighter-rouge">sonarqube-token</code></li>
</ol>

<p><img src="https://arshadmehmood.com/assets/images/posts/sonarqube-jenkins.png" alt="SonarQube Jenkins Integration" /></p>

<h3 id="configure-sonarqube-scanner">Configure SonarQube Scanner</h3>

<ol>
  <li>Go to Manage Jenkins → Global Tool Configuration</li>
  <li>Scroll to “SonarQube Scanner”</li>
  <li>Click “Add SonarQube Scanner”</li>
  <li>Name: <code class="language-plaintext highlighter-rouge">SonarQube Scanner</code></li>
  <li>Install automatically from Maven Central</li>
</ol>

<h3 id="create-pipeline-with-sonarqube-analysis">Create Pipeline with SonarQube Analysis</h3>

<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">pipeline</span> <span class="o">{</span>
    <span class="n">agent</span> <span class="n">any</span>
    
    <span class="n">environment</span> <span class="o">{</span>
        <span class="n">SCANNER_HOME</span> <span class="o">=</span> <span class="n">tool</span> <span class="s1">'SonarQube Scanner'</span>
    <span class="o">}</span>
    
    <span class="n">stages</span> <span class="o">{</span>
        <span class="n">stage</span><span class="o">(</span><span class="s1">'Checkout'</span><span class="o">)</span> <span class="o">{</span>
            <span class="n">steps</span> <span class="o">{</span>
                <span class="n">checkout</span> <span class="n">scm</span>
            <span class="o">}</span>
        <span class="o">}</span>
        
        <span class="n">stage</span><span class="o">(</span><span class="s1">'SonarQube Analysis'</span><span class="o">)</span> <span class="o">{</span>
            <span class="n">steps</span> <span class="o">{</span>
                <span class="n">withSonarQubeEnv</span><span class="o">(</span><span class="s1">'SonarQube'</span><span class="o">)</span> <span class="o">{</span>
                    <span class="n">sh</span> <span class="s1">'''
                        $SCANNER_HOME/bin/sonar-scanner \
                        -Dsonar.projectKey=my-project \
                        -Dsonar.sources=. \
                        -Dsonar.host.url=http://localhost:9000 \
                        -Dsonar.login=$SONAR_AUTH_TOKEN
                    '''</span>
                <span class="o">}</span>
            <span class="o">}</span>
        <span class="o">}</span>
        
        <span class="n">stage</span><span class="o">(</span><span class="s1">'Quality Gate'</span><span class="o">)</span> <span class="o">{</span>
            <span class="n">steps</span> <span class="o">{</span>
                <span class="n">timeout</span><span class="o">(</span><span class="nl">time:</span> <span class="mi">1</span><span class="o">,</span> <span class="nl">unit:</span> <span class="s1">'HOURS'</span><span class="o">)</span> <span class="o">{</span>
                    <span class="n">waitForQualityGate</span> <span class="nl">abortPipeline:</span> <span class="kc">true</span>
                <span class="o">}</span>
            <span class="o">}</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<h3 id="project-specific-configuration">Project-specific Configuration</h3>

<p>Create <code class="language-plaintext highlighter-rouge">sonar-project.properties</code> in your project root:</p>

<div class="language-properties highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="py">sonar.projectKey</span><span class="p">=</span><span class="s">my-project-key</span>
<span class="py">sonar.projectName</span><span class="p">=</span><span class="s">My Project</span>
<span class="py">sonar.projectVersion</span><span class="p">=</span><span class="s">1.0</span>
<span class="py">sonar.sources</span><span class="p">=</span><span class="s">src</span>
<span class="py">sonar.language</span><span class="p">=</span><span class="s">java</span>
<span class="py">sonar.sourceEncoding</span><span class="p">=</span><span class="s">UTF-8</span>
</code></pre></div></div>

<h3 id="quality-gates">Quality Gates</h3>

<p>SonarQube Quality Gates ensure code meets quality standards before deployment. Configure thresholds for:</p>

<ul>
  <li>Code coverage</li>
  <li>Duplicated lines</li>
  <li>Maintainability rating</li>
  <li>Reliability rating</li>
  <li>Security rating</li>
</ul>

<p>The pipeline will fail if quality gate conditions aren’t met, preventing poor quality code from being deployed.</p>]]></content><author><name>Arshad Mehmood</name></author><category term="DevOps" /><category term="devops" /><category term="sonarqube" /><category term="jenkins" /><category term="code-quality" /><category term="automation" /><summary type="html"><![CDATA[Note This post is part of the DevOps Journey]]></summary></entry><entry><title type="html">Setup Dynamic Jenkins Agents in Kubernetes</title><link href="https://arshadmehmood.com/devops/setup-dynamic-jenkins-agents-kubernetes/" rel="alternate" type="text/html" title="Setup Dynamic Jenkins Agents in Kubernetes" /><published>2024-10-10T00:00:00+00:00</published><updated>2024-10-10T00:00:00+00:00</updated><id>https://arshadmehmood.com/devops/setup-dynamic-jenkins-agents-kubernetes</id><content type="html" xml:base="https://arshadmehmood.com/devops/setup-dynamic-jenkins-agents-kubernetes/"><![CDATA[<p><em><strong>Note</strong> This post is part of the <a href="/software/devops-journey/">DevOps Journey</a></em></p>

<p>We want to use a Kubernetes Cluster with Jenkins, so that Jenkins can fire up slaves in the cluster as required and perform the pipeline tasks.
Instead of paying a lot of money for the Kubernetes Cluster, we will set it up locally using <a href="https://kind.sigs.k8s.io/">Kind</a>.</p>

<h3 id="kubernetes-kind">Kubernetes Kind</h3>

<h4 id="install">Install</h4>
<p>Install <a href="https://kind.sigs.k8s.io/">Kind</a> with Homebrew:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brew <span class="nb">install </span>kind
</code></pre></div></div>

<h4 id="setup-the-cluster">Setup the Cluster</h4>

<p>Create a <code class="language-plaintext highlighter-rouge">yaml</code> config file called <code class="language-plaintext highlighter-rouge">kind-config.yaml</code> for kind:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">kind</span><span class="pi">:</span> <span class="s">Cluster</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">kind.x-k8s.io/v1alpha4</span>
<span class="na">networking</span><span class="pi">:</span>
  <span class="c1"># WARNING: It is _strongly_ recommended that you keep this the default</span>
  <span class="c1"># (127.0.0.1) for security reasons. However it is possible to change this.</span>
  <span class="na">apiServerAddress</span><span class="pi">:</span> <span class="s2">"</span><span class="s">127.0.0.1"</span>
  <span class="c1"># By default the API server listens on a random open port.</span>
  <span class="c1"># You may choose a specific port but probably don't need to in most cases.</span>
  <span class="c1"># Using a random port makes it easier to spin up multiple clusters.</span>
  <span class="na">apiServerPort</span><span class="pi">:</span> <span class="m">53850</span>
</code></pre></div></div>

<p>Create the cluster with <code class="language-plaintext highlighter-rouge">kind-config.yaml</code> file:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kind create cluster <span class="nt">--config</span> kind-config.yaml
</code></pre></div></div>

<p>Verify the cluster:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl cluster-info <span class="nt">--context</span> kind-kind
</code></pre></div></div>

<p>In Kubernetes, namespaces provide a mechanism for isolating groups of resources within a single cluster.
Create Namespace <code class="language-plaintext highlighter-rouge">jenkins</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl create namespace jenkins
</code></pre></div></div>

<p>A ServiceAccount will allow Jenkins to spin up the required slave pods as needed.
Create a ServiceAccount named <code class="language-plaintext highlighter-rouge">jenkins</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl create serviceaccount jenkins <span class="nt">--namespace</span><span class="o">=</span>jenkins
</code></pre></div></div>

<p>Create a token for the ServiceAccount created above, this token will be used in the Jenkins for authentication:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl create token jenkins <span class="nt">--duration</span><span class="o">=</span>999999h <span class="nt">--namespace</span><span class="o">=</span>jenkins
</code></pre></div></div>
<p><img src="https://arshadmehmood.com/assets/images/posts/k8s-token.png" alt="Token" />
Copy this token, we will use it later in Jenkins.</p>

<p>Create admin role for the ServiceAccount:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl create rolebinding jenkins-admin-binding <span class="nt">--clusterrole</span><span class="o">=</span>admin <span class="nt">--serviceaccount</span><span class="o">=</span>jenkins:jenkins <span class="nt">--namespace</span><span class="o">=</span>jenkins
</code></pre></div></div>

<p>Kubernetes cluster is setup to be used in Jenkins now.</p>

<h3 id="jenkins-setup">Jenkins Setup</h3>

<p>Now we’ll configure Jenkins to use our Kubernetes cluster for dynamic agent provisioning.</p>

<h4 id="install-the-kubernetes-plugin">Install the Kubernetes Plugin</h4>

<ol>
  <li>Go to Jenkins Dashboard → Manage Jenkins → Manage Plugins</li>
  <li>Search for “Kubernetes” plugin and install it</li>
  <li>Restart Jenkins if required</li>
</ol>

<p><img src="https://arshadmehmood.com/assets/images/posts/k8s-plugin-jenkins.png" alt="Kubernetes Plugin Installation" /></p>

<h4 id="configure-kubernetes-cloud">Configure Kubernetes Cloud</h4>

<ol>
  <li>Navigate to Manage Jenkins → Configure System</li>
  <li>Scroll down to “Cloud” section and click “Add a new cloud” → “Kubernetes”</li>
  <li>Configure the following settings:</li>
</ol>

<p><strong>Kubernetes Cloud Details:</strong></p>
<ul>
  <li>Name: <code class="language-plaintext highlighter-rouge">kubernetes</code></li>
  <li>Kubernetes URL: <code class="language-plaintext highlighter-rouge">https://127.0.0.1:53850</code> (from our Kind cluster)</li>
  <li>Kubernetes Namespace: <code class="language-plaintext highlighter-rouge">jenkins</code></li>
  <li>Credentials: Add → Jenkins → Secret text
    <ul>
      <li>Secret: (paste the token we created earlier)</li>
      <li>ID: <code class="language-plaintext highlighter-rouge">kubernetes-token</code></li>
      <li>Description: <code class="language-plaintext highlighter-rouge">Kubernetes ServiceAccount Token</code></li>
    </ul>
  </li>
</ul>

<p><img src="https://arshadmehmood.com/assets/images/posts/jenkins-credentials.png" alt="Jenkins Kubernetes Credentials" /></p>

<p><strong>Test Connection:</strong> Click “Test Connection” to verify the setup.</p>

<p><img src="https://arshadmehmood.com/assets/images/posts/test-connection-jenkins-k8s.png" alt="Test Kubernetes Connection" /></p>

<h4 id="create-pod-template">Create Pod Template</h4>

<p>In the same Kubernetes cloud configuration:</p>

<ol>
  <li>Scroll down to “Pod Templates” and click “Add Pod Template”</li>
  <li>Configure:
    <ul>
      <li>Name: <code class="language-plaintext highlighter-rouge">jenkins-agent</code></li>
      <li>Namespace: <code class="language-plaintext highlighter-rouge">jenkins</code></li>
      <li>Labels: <code class="language-plaintext highlighter-rouge">jenkins-agent</code></li>
      <li>Usage: “Use this node as much as possible”</li>
    </ul>
  </li>
</ol>

<p><strong>Container Template:</strong></p>
<ul>
  <li>Name: <code class="language-plaintext highlighter-rouge">jnlp</code></li>
  <li>Docker Image: <code class="language-plaintext highlighter-rouge">jenkins/inbound-agent:latest</code></li>
  <li>Working Directory: <code class="language-plaintext highlighter-rouge">/home/jenkins/agent</code></li>
  <li>Command to run: (leave empty)</li>
  <li>Arguments to pass: (leave empty)</li>
</ul>

<p><strong>Volume Mounts:</strong> Add if needed for Docker-in-Docker or specific tools.</p>

<p><img src="https://arshadmehmood.com/assets/images/posts/jenks-k8s-cloud.png" alt="Jenkins Kubernetes Cloud Configuration" /></p>

<h4 id="test-dynamic-agent">Test Dynamic Agent</h4>

<p>Create a simple pipeline to test:</p>

<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">pipeline</span> <span class="o">{</span>
    <span class="n">agent</span> <span class="o">{</span>
        <span class="n">kubernetes</span> <span class="o">{</span>
            <span class="n">label</span> <span class="s1">'jenkins-agent'</span>
        <span class="o">}</span>
    <span class="o">}</span>
    <span class="n">stages</span> <span class="o">{</span>
        <span class="n">stage</span><span class="o">(</span><span class="s1">'Test'</span><span class="o">)</span> <span class="o">{</span>
            <span class="n">steps</span> <span class="o">{</span>
                <span class="n">sh</span> <span class="s1">'echo "Running on Kubernetes agent"'</span>
                <span class="n">sh</span> <span class="s1">'kubectl version --client'</span>
            <span class="o">}</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>

<p>The Jenkins agent pod will be created automatically in the Kubernetes cluster when the pipeline runs and destroyed after completion.</p>

<h3 id="verification">Verification</h3>

<ul>
  <li>Check pods in the jenkins namespace: <code class="language-plaintext highlighter-rouge">kubectl get pods -n jenkins</code></li>
  <li>Monitor Jenkins logs for any connection issues</li>
  <li>Verify agents are created and destroyed dynamically</li>
</ul>

<p>This setup provides scalable, on-demand Jenkins agents using Kubernetes resources efficiently.</p>]]></content><author><name>Arshad Mehmood</name></author><category term="DevOps" /><category term="devops" /><category term="kubernetes" /><category term="jenkins" /><category term="automation" /><category term="ci-cd" /><summary type="html"><![CDATA[Learn how to configure Jenkins to dynamically create agents in a Kubernetes cluster using Kind. Step-by-step guide to scalable CI/CD infrastructure.]]></summary></entry></feed>