<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://pierre-couy.dev/feed.xml" rel="self" type="application/atom+xml" /><link href="https://pierre-couy.dev/" rel="alternate" type="text/html" /><updated>2026-02-26T11:01:25+00:00</updated><id>https://pierre-couy.dev/feed.xml</id><title type="html">Pierre Couy’s tech corner</title><subtitle>Freelance developper, computer science teacher, passionate about programming. I enjoy sharing my work with the world.</subtitle><author><name>Pierre Couy</name></author><entry><title type="html">AI learns to play Mario : Deep Reinforcement Learning applied to Super Mario Bros</title><link href="https://pierre-couy.dev/tinkering/2026/02/training-reinforcement-learning-mario-bros.html" rel="alternate" type="text/html" title="AI learns to play Mario : Deep Reinforcement Learning applied to Super Mario Bros" /><published>2026-02-23T00:00:00+00:00</published><updated>2026-02-23T00:00:00+00:00</updated><id>https://pierre-couy.dev/tinkering/2026/02/training-reinforcement-learning-mario-bros</id><content type="html" xml:base="https://pierre-couy.dev/tinkering/2026/02/training-reinforcement-learning-mario-bros.html"><![CDATA[<!-- Add a placeholder for the Twitch embed -->
<div id="twitch-embed"></div>
<!-- Load the Twitch embed JavaScript file -->
<script src="https://embed.twitch.tv/embed/v1.js"></script>

<!-- Create a Twitch.Embed object that will render within the "twitch-embed" element -->
<script type="text/javascript">
  new Twitch.Player("twitch-embed", {
    width: 854,
    height: 480,
    channel: "pcouy_",
    // Only needed if this page is going to be embedded on other websites
    //parent: ["embed.example.com", "othersite.example.com"]
  });
</script>

<p>This is a live feed of my Deep Reinforcement Learning agent training on the
original Super Mario Bros game.</p>

<p>The agent is a custom implementation of the <a href="https://arxiv.org/abs/1710.02298">Rainbow DQN
paper</a> from 2017. It stays pretty close to
what’s described in the papers. There are however a few custom tweaks. I’ll
update this article soon with more details.</p>

<p>This originally started as a pedagogical implementation of the original <a href="https://arxiv.org/abs/1312.5602">Deep Q
Network paper</a> from 2015 for a course I taught.
I then kept implementing improvements to this paper until I reached the
algorithm described in the Rainbow paper.</p>]]></content><author><name>Pierre Couy</name></author><category term="Tinkering" /><category term="AI" /><category term="reinforcement learning" /><category term="simulation" /><category term="emergence" /><category term="scientific programming" /><summary type="html"><![CDATA[I reimplemented Deep Reinforcement learning research papers and training an AI to beat Super Mario Bros]]></summary></entry><entry><title type="html">Mitosis in the Gray-Scott model : writing shader-based chemical simulations</title><link href="https://pierre-couy.dev/simulations/2024/09/gray-scott-shader.html" rel="alternate" type="text/html" title="Mitosis in the Gray-Scott model : writing shader-based chemical simulations" /><published>2024-09-08T00:00:00+00:00</published><updated>2024-09-08T00:00:00+00:00</updated><id>https://pierre-couy.dev/simulations/2024/09/gray-scott-shader</id><content type="html" xml:base="https://pierre-couy.dev/simulations/2024/09/gray-scott-shader.html"><![CDATA[<p>The <a href="https://groups.csail.mit.edu/mac/projects/amorphous/GrayScott/">Gray Scott Model of Reaction Diffusion</a>
is an interesting instance of <a href="https://en.wikipedia.org/wiki/Emergence">emergence</a>.
By simulating a small chemical system that involves only a few components and
reactions, complex and mesmerizing patterns appear.</p>

<figure>
<iframe srcdoc="&lt;div style=&quot;position:absolute;top:33%;color:white;&quot;&gt;Please be patient as shadertoy is overloaded sometimes. While you wait for shadertoy to load, you can watch the video &lt;a target=&quot;_parent&quot; href=&quot;https://pierre-couy.dev/simulations/2024/09/gray-scott-shader.html#interesting-emergent-behaviors&quot;&gt; at the end of the article&lt;/a&gt;&lt;/div&gt;" onload="this.removeAttribute('srcdoc')" src="https://www.shadertoy.com/embed/lXXcz7?gui=true&amp;t=0&amp;paused=false&amp;muted=false" width="640" height="360" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
    <legend>Since ShaderToy embeds are often broken, you can <a href="https://www.shadertoy.com/view/lXXcz7">run it directly from their
    website</a>
    </legend>
</figure>

<p>You can interact with the simulation above by clicking on it to drop some green
and you can reset it by pressing the previous (⏮️) button.</p>

<p>Although the local rules and the underlying math are quite simple, there is some
heavy computations involved. For each time step in the simulation, we must
apply these rules to compute the concentrations of every involved component
at every possible location. Running such a simulation on a CPU would be
extremely slow. GPUs, however, are specifically built to handle large volumes of
a single small computation in parallel.</p>

<p>This post is an introduction to writing such simulations using <a href="https://www.khronos.org/registry/OpenGL/specs/es/3.0/GLSL_ES_Specification_3.00.pdf">GLSL
ES</a>,
with a <a href="https://www.shadertoy.com/view/lXXcz7">basic implementation of the Gray Scott model that runs in the browser on
Shadertoy</a> that is less than 100 lines of
code.</p>

<ol id="markdown-toc">
  <li><a href="#prerequisites" id="markdown-toc-prerequisites">Prerequisites</a>    <ol>
      <li><a href="#computing-simulations" id="markdown-toc-computing-simulations">Computing simulations</a></li>
      <li><a href="#gray-scott-model" id="markdown-toc-gray-scott-model">Gray Scott model</a>        <ol>
          <li><a href="#chemical-reactions" id="markdown-toc-chemical-reactions">Chemical reactions</a></li>
          <li><a href="#diffusion" id="markdown-toc-diffusion">Diffusion</a></li>
          <li><a href="#catalytic-reactions" id="markdown-toc-catalytic-reactions">Catalytic reactions</a></li>
          <li><a href="#auto-catalytic-reactions" id="markdown-toc-auto-catalytic-reactions">Auto-catalytic reactions</a></li>
          <li><a href="#the-gray-scott-model-itself" id="markdown-toc-the-gray-scott-model-itself">The Gray-Scott model itself</a></li>
        </ol>
      </li>
      <li><a href="#shaders" id="markdown-toc-shaders">Shaders</a>        <ol>
          <li><a href="#what-is-a-shader-" id="markdown-toc-what-is-a-shader-">What is a shader ?</a></li>
          <li><a href="#the-basics-of-writing-shaders" id="markdown-toc-the-basics-of-writing-shaders">The basics of writing shaders</a></li>
        </ol>
      </li>
    </ol>
  </li>
  <li><a href="#implementing-the-simulation" id="markdown-toc-implementing-the-simulation">Implementing the simulation</a>    <ol>
      <li><a href="#update-rules" id="markdown-toc-update-rules">Update rules</a>        <ol>
          <li><a href="#reactions" id="markdown-toc-reactions">Reactions</a>            <ol>
              <li><a href="#auto-catalytic-reactions-1" id="markdown-toc-auto-catalytic-reactions-1">Auto-catalytic reactions</a></li>
              <li><a href="#gray-scott-reactions" id="markdown-toc-gray-scott-reactions">Gray-Scott reactions</a></li>
            </ol>
          </li>
          <li><a href="#diffusion-1" id="markdown-toc-diffusion-1">Diffusion</a></li>
        </ol>
      </li>
      <li><a href="#visual-representation-of-the-system" id="markdown-toc-visual-representation-of-the-system">Visual representation of the system</a></li>
      <li><a href="#texture-buffer" id="markdown-toc-texture-buffer">Texture buffer</a></li>
      <li><a href="#simulation-shader" id="markdown-toc-simulation-shader">Simulation shader</a>        <ol>
          <li><a href="#initialization" id="markdown-toc-initialization">Initialization</a></li>
          <li><a href="#update-rules-implementation" id="markdown-toc-update-rules-implementation">Update rules implementation</a></li>
          <li><a href="#final-shader" id="markdown-toc-final-shader">Final shader</a></li>
        </ol>
      </li>
    </ol>
  </li>
  <li><a href="#playing-with-the-simulation" id="markdown-toc-playing-with-the-simulation">Playing with the simulation</a>    <ol>
      <li><a href="#interesting-emergent-behaviors" id="markdown-toc-interesting-emergent-behaviors">Interesting emergent behaviors</a></li>
      <li><a href="#hacking-on-the-code" id="markdown-toc-hacking-on-the-code">Hacking on the code</a></li>
      <li><a href="#continuous-cellular-automata" id="markdown-toc-continuous-cellular-automata">Continuous cellular automata</a></li>
    </ol>
  </li>
  <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
</ol>

<h2 id="prerequisites">Prerequisites</h2>

<p>In this section, I’ll try to quickly introduce some important concepts in a
short and beginner friendly way.</p>

<h3 id="computing-simulations">Computing simulations</h3>

<p>Simulating any kind of physical system involves computing what happens at any
possible location, for any possible moment in time.</p>

<p>However, the world we live our daily lives in is <em>continuous</em> in regards to space and
time : the real world is not made in a voxel grid of even the smallest size.
Likewise, even the shortest durations can still be split into smaller
durations. Worded differently, any volume larger than 0 contains an infinite
amount of points in space, and any duration larger than 0 contains an infinite amount of
points in time.</p>

<p>Computers cannot simulate a continuous world because it would require infinite
computations to handle even the tiniest fractions of space and time. To overcome
this, we will <a href="https://en.wikipedia.org/wiki/Discretization"><em>discretize</em></a> both
space and time.</p>

<p>Discretization is the action of subdividing space into a fixed grid, and time into
fixed elementary durations (<em>time steps</em>). For each cell of this grid, we will repeatedly
run a computation to determine how its content changes over one time step. This
results in an approximation of a continuous world. The smaller our grid and time
steps, the more accurate our simulation.</p>

<p>Since we will be using shaders, which is a technology from computer graphics, it
makes sense to use pixels as grid cells, and frames (as in <em>frames per second</em>)
as elementary time steps. (The simulation we will build will run in a 2D space
for easier visualization and manageable computations).</p>

<p>In the following, I’ll use \(dT\) to represent the duration of an elementary time step.
\(dX\) and \(dY\) will be the size of a single grid cell. Elementary lengths and durations
in discrete spaces are usually written with the \(\Delta\) prefix, but using
\(d\) instead will make it consistent with the code.</p>

<h3 id="gray-scott-model">Gray Scott model</h3>

<script id="MathJax-script" async="" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>

<p>The Gray Scott model describes a specific family of <em>reaction-diffusion</em> systems.
More specifically, they involve an <em>auto-catalytic</em> reaction.
Let’s first define and mathematically describe these terms. This will allow us
to derive general update rules for reaction-diffusion models.
Then I will describe what makes the Gray-Scott model interesting as a specific
instance of reaction-diffusion.</p>

<h4 id="chemical-reactions">Chemical reactions</h4>

<p>A chemical <em>reaction</em> is the process in which one or more chemical species
(reactants) are consumed to produce one or more other chemical species
(products).</p>

<p>Chemical reactions are usually described with an equation that summarizes their
outcome, such as the following :</p>

<p>$$ A + 2B \rightarrow 4C $$</p>

<p>In this example, the reaction that is described produces 4 molecules of the \(C\)
chemical species by consuming 1 molecule of \(A\) and 2 molecules of \(B\).</p>

<p>The speed at which a chemical reaction occurs is defined as the quantity of
molecules that get transformed in a given amount of time.
Every reactant (molecule that is listed on the left side of the equation) need to
meet at the same time and spot for the reaction to happen. Since the probability
of finding a molecule at any given spot is proportional to its concentration,
the speed of the reaction is proportional to the concentration of every
reactant.</p>

<p>The speed of the example reaction above is then :</p>

<p>$$ speed = k*[A]*[B]*[B]  $$</p>

<p>where \([X]\) is the concentration of molecule \(X\), and \(k\) is a positive constant
we’ll call the “speed constant of the reaction”. Since 2 molecules of \(B\) are
required at the input of the reaction, it needs to appear twice in the formula
for the reaction speed.</p>

<h4 id="diffusion">Diffusion</h4>

<p>Diffusion is the process through which certain quantities, such as chemical
species concentrations, tend to spread out and homogenize. It can be easily
observed by putting a drop of ink into a glass of water.</p>

<p>One way to think about it in the context of the simulation is that each cell of
the 2D grid is slowly and constantly leaking a fixed proportion of its content
to its neighbors. At the same time, it’s receiving content leaked from its
neighbouring cells. We will write \(\tau_X\) as this fixed proportion for chemical
species \(X\) that leaks over a base unit of time, and we will call it the diffusion rate.</p>

<p>Let’s consider two simple cases to make sure this is a reasonable way to model
diffusion :</p>

<ul>
  <li>If the quantity is homogeneous over the grid (the same in every cell), the outgoing
amount will be the same as the incoming one, resulting in the quantity staying
homogeneous.</li>
  <li>On the other hand, if a cell \(C_1\) contains more than a neighbor cell \(C_2\), then
\(C_1\) will leak a larger amount to its neighbors than \(C_2\). This will cause
the quantity inside \(C_1\) to decrease, while the quantity inside \(C_2\) will
increase. In the end, quantities inside \(C_1\) and \(C_2\) are closer together
than they were at the start, which is consistent for a process that homogenizes
quantities.</li>
</ul>

<h4 id="catalytic-reactions">Catalytic reactions</h4>

<p>Some chemical reactions require a specific chemical species, which does not
seem to get consumed by the reaction, to be present for the reaction to
happen. Let’s consider a very simple reaction where a species \(A\) transforms to
species \(B\) :</p>

<p>$$A \rightarrow B$$</p>

<p>If this reaction only happens when a third species \(C\) is present, we say that
\(C\) is a <em>catalyst</em> for this reaction, and the reaction is said to be
<em>catalytic</em>. While the outcome of the reaction does
not affect the concentration in species \(C\), we can still make it appear in
the reaction’s equation to account for its role. We do this by making it appear
on both sides of the arrow :</p>

<p>$$A + C \rightarrow B + C$$</p>

<p>This changes the formula for speed of the reaction to :</p>

<p>$$speed = k * [A] * [C]$$</p>

<p>If \(C\) is absent, then \([C]=0\), making the speed 0 as well. This
is consistent with \(C\) being a catalyst for the reaction. This also implies that
while the concentration in \(C\) does not change during the reaction, the speed
of the reaction is proportional to the concentration of species \(C\).</p>

<h4 id="auto-catalytic-reactions">Auto-catalytic reactions</h4>

<p>Among the family of catalytic reactions, there are special cases where the
catalyst is also a product of the reaction : the reaction requires the catalyst
to be present in order to happen and produces more of the catalyst in turn.
Such reactions are called <a href="https://en.wikipedia.org/wiki/Autocatalysis"><em>auto-catalytic</em>
reactions</a>. The equation still
makes the catalyst appear on both sides, but in this case, the number in front
of it will be larger on the right side, indicating an increase in concentration
for the catalyst. For instance :</p>

<p>$$ 2A + C \rightarrow 2C $$</p>

<p>Autocatalytic reactions are of special interest because of their role in biology and
their <a href="https://en.wikipedia.org/wiki/Abiogenesis">supposed role in the origin of life</a>.</p>

<h4 id="the-gray-scott-model-itself">The Gray-Scott model itself</h4>

<p>I previously referred to the Gray-Scott model as a “Reaction-Diffusion model for
a specific auto-catalytic reaction”. This means that in this model, both
diffusion and an auto-catalytic reaction will happen simultaneously. The main
reaction involves two chemical species \(A\) and \(B\) which react according to
the following equation :</p>

<p>\(A + 2B \rightarrow 3B\) with speed constant \(S\)</p>

<p>We also consider two hypothetical reactions :</p>

<ul>
  <li>\(\emptyset \rightarrow A\) which constantly adds species \(A\) at rate \(F\)</li>
  <li>\(B \rightarrow \emptyset\) which removes \(B\) with speed constant \(K\)</li>
</ul>

<p>We call \(F\) the <em>feed rate</em> and \(K\) is the <em>kill rate</em>. If we look at the
main reaction as \(A\) being used as food by \(B\), \(F\) is indeed the rate at
which we add food to the system, and \(K\) is the rate at which we remove – or
kill – \(B\).</p>

<p>Let’s also add a process that removes a fixed fraction of \(A\) and \(B\) from
every cell at each time step. We will use the value of the feed rate as a speed
constant for this process.</p>

<p>Diffusion, occurring concurrently with these reactions, is crucial for complex
patterns to emerge. It will allow \(B\) to
propagate through space, starting the autocatalytic reaction in new grid cells
which did not previously contain any of the chemical \(B\).
Diffusion will also let \(A\) flow from regions where it is more abundant to
regions where it is rarer (because it was consumed by \(B\)).</p>

<p>By tuning the relative values of \(S\), \(F\), \(K\), \(\tau_A\) and \(\tau_B\),
different kind of complex patterns can emerge.</p>

<h3 id="shaders">Shaders</h3>

<p>Shaders are a specific kind of computer programs that are designed to run on a
Graphical Processing Unit (GPU). They are generally written in a dedicated
programming language and are mostly used to control how a 3D world is rendered
to a 2D screen.</p>

<p>Since <a href="https://www.youtube.com/watch?v=0ifChJ0nJfM">good resources</a> on
<a href="https://thebookofshaders.com/">how to start writing shaders</a> already exist, this
introduction will focus on the most relevant parts for implementing a Gray-Scott
simulation.</p>

<h4 id="what-is-a-shader-">What is a shader ?</h4>

<p>Most shaders come in one of two flavors : <em>vertex shaders</em> and <em>fragment
shaders</em>. Rendering a 3D world to a 2D screen roughly involves two steps :</p>

<ul>
  <li>Convert coordinates from the 3D space to a position on screen. This
involves performing a <a href="https://en.wikipedia.org/wiki/3D_projection">projection</a>
depending on the position of the camera. Vertex shaders perform this step</li>
  <li>Determine the color of each fragment (pixel) on the screen. This involves
running computations on the output of the previous step for every pixel of the
screen, which is what fragment shaders are made for.</li>
</ul>

<p>Running a computation for every pixel should remind you of how we derived update
rules that should be applied to every cell of a grid. By implementing such a
simulation as a fragment shader, we will take advantage of the main strength of
GPUs : parallelization.</p>

<p>While there surely exists a better way to run simulations on the GPU, the amount
of resources dedicated to learning computer graphics made it a lot easier (to
me) to start writing and running code on the GPU. There are even web-based
editors (such as <a href="https://www.shadertoy.com/">Shadertoy</a> which I used for this)
that let you compile and run your shaders in the browser without having to
install anything. As a bonus, this is agnostic of the GPU brand (by contrast with Cuda
which is Nvidia specific, or ROCm for AMD).</p>

<h4 id="the-basics-of-writing-shaders">The basics of writing shaders</h4>

<p>As I am not a shader expert, I will focus on how to get one running on
Shadertoy. My understanding is that there is a lot of boilerplate that Shadertoy
handles and that it is the most straightforward way to begin writing shaders.</p>

<p>GLSL, like most programming languages, uses variables, functions, conditionals
and loops.
Assuming you already used a few different programming languages, you will
probably be comfortable with reading and tinkering with shader code. However,
there are some specificities of <em>OpenGL Shading Language</em> (GLSL) that may surprise you.</p>

<p>The most important thing to understand is that we are going to write a function
that will be run on the GPU for every pixel of every frame. This function will
take the coordinates of a pixel as an input, and will output the color that
should be used for this pixel. For our simulation, this means that this function
will be in charge of simulating one grid cell for one time step. It will then be
repeatedly executed, resulting in the full animated simulation.</p>

<p>Using Shadertoy, this function will look like the following :</p>

<div class="language-glsl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">mainImage</span><span class="p">(</span> <span class="k">out</span> <span class="kt">vec4</span> <span class="n">fragColor</span><span class="p">,</span> <span class="k">in</span> <span class="kt">vec2</span> <span class="n">fragCoord</span> <span class="p">)</span>
<span class="p">{</span>
    <span class="c1">// Normalized pixel coordinates (from 0 to 1)</span>
    <span class="kt">vec2</span> <span class="n">uv</span> <span class="o">=</span> <span class="n">fragCoord</span><span class="o">/</span><span class="n">iResolution</span><span class="p">.</span><span class="n">xy</span><span class="p">;</span>

    <span class="c1">// Time and space varying pixel color</span>
    <span class="kt">vec3</span> <span class="n">col</span> <span class="o">=</span> <span class="mi">0</span><span class="p">.</span><span class="mi">5</span> <span class="o">+</span> <span class="mi">0</span><span class="p">.</span><span class="mi">5</span><span class="o">*</span><span class="n">cos</span><span class="p">(</span><span class="n">iTime</span><span class="o">+</span><span class="n">uv</span><span class="p">.</span><span class="n">xyx</span><span class="o">+</span><span class="kt">vec3</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">4</span><span class="p">));</span>

    <span class="c1">// Output to screen</span>
    <span class="n">fragColor</span> <span class="o">=</span> <span class="kt">vec4</span><span class="p">(</span><span class="n">col</span><span class="p">,</span><span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>There are a few things to notice already. First, this function does not return
anything. Instead, the pixel’s color is output by setting the value of
<code class="language-plaintext highlighter-rouge">fragColor</code>, which is defined as an <code class="language-plaintext highlighter-rouge">out</code> parameter of the function.</p>

<p>You may also notice the use of variables which were not previously defined, such as
<code class="language-plaintext highlighter-rouge">iResolution</code> or <code class="language-plaintext highlighter-rouge">iTime</code>. These variables are called <em>uniforms</em>, and their
values are provided from the outside of the shader. This is part of the
boilerplate that Shadertoy handles for us.</p>

<p>Shaders usually make heavy use of vectors, which can have 2, 3 or 4 dimensions.
The sample code above features the <code class="language-plaintext highlighter-rouge">vec*</code> types for all three sizes of vectors.
GLSL comes with a convenient syntax for picking and rearranging vector
components. If we have <code class="language-plaintext highlighter-rouge">x = vec4(0.0, 0.2, 0.4, 0.6)</code>, then <code class="language-plaintext highlighter-rouge">x.xyy</code> will be
equal to <code class="language-plaintext highlighter-rouge">vec3(0.0, 0.2, 0.2)</code>. You can reference each vector coordinate by <code class="language-plaintext highlighter-rouge">x</code>,
<code class="language-plaintext highlighter-rouge">y</code>, <code class="language-plaintext highlighter-rouge">z</code> and <code class="language-plaintext highlighter-rouge">w</code> respectively. Since vectors are also used to represent colors,
the symbols <code class="language-plaintext highlighter-rouge">r</code>, <code class="language-plaintext highlighter-rouge">g</code>, <code class="language-plaintext highlighter-rouge">b</code> and <code class="language-plaintext highlighter-rouge">a</code> can also be used.</p>

<h2 id="implementing-the-simulation">Implementing the simulation</h2>

<h3 id="update-rules">Update rules</h3>

<p>This is the most math-heavy section of the article, in which we derive the
update rules for the simulation.</p>

<h4 id="reactions">Reactions</h4>

<p>In the general case, when simulating the reaction \(A + 2B \rightarrow 4C\) for a single time step, the
concentrations of \(A\), \(B\) and \(C\) need to be updated in the following way
for every \((x,y)\) cell in the simulation :</p>

<p>
$$ [A](t+dT,x,y) = [A](t,x,y) - speed(t,x,y)*dT $$
$$ [B](t+dT,x,y) = [B](t,x,y) - 2*speed(t,x,y)*dT $$
$$ [C](t+dT,x,y) = [C](t,x,y) + 4*speed(t,x,y)*dT $$
</p>

<p>Replacing \(speed(t,x,y)\) with its expression from above yields :</p>

<p>
$$ [A](t+dT,x,y) = [A](t,x,y) - k*[A](t,x,y)*[B](t,x,y)^2*dT $$
$$ [B](t+dT,x,y) = [B](t,x,y) - 2*k*[A](t,x,y)*[B](t,x,y)^2*dT $$
$$ [C](t+dT,x,y) = [C](t,x,y) + 4*k*[A](t,x,y)*[B](t,x,y)^2*dT $$
</p>

<p>where \( dT \) is the duration of a time step. Notice that the numbers in
front of \(speed(t)\) come from the quantities in the equation that summarizes
the reaction.</p>

<p>Looking closely at this update rule, you may notice that chemical reactions
happen independently in each cell. This can be evidenced by the fact that the
formula for updating cell \((x,y)\) only involves coordinates \((x,y)\) and
ignores concentrations in neighboring cells (such as \((x+dX,y)\)).</p>

<h5 id="auto-catalytic-reactions-1">Auto-catalytic reactions</h5>

<p>Consider the following auto-catalytic reaction :</p>

<p>
$$ 2A + C \rightarrow 2C $$
</p>

<p>The update rule for this reaction is then</p>

<p>
$$ [A](t+dT,x,y) = [A](t,x,y) - 2*k*[A](t,x,y)^2*[C](t,x,y)*dT $$
$$ [C](t+dT,x,y) = [C](t,x,y) + (2-1)*k*[A](t,x,y)^2*[C](t,x,y)*dT $$
</p>

<h5 id="gray-scott-reactions">Gray-Scott reactions</h5>

<p>By combining the update rules for all processes (which consists in
successively applying them) previously described, we get the following
update rules for the “reactions” part of our Gray-Scott model :</p>

<p>
$$[A](t+dT) = [A](t) + (F - S * [A](t) * [B](t)^2 - F * [A](t)) * dT$$
$$[B](t+dT) = [B](t) + (S * [A](t) * [B](t)^2 - K * [B](t) - F * [B](t)) * dT $$
</p>

<p>which can be rearranged as :</p>

<p>
$$[A](t+dT) = [A](t) + (F * (1-[A](t)) - S * [A](t) * [B](t)^2) * dT$$
$$[B](t+dT) = [B](t) + (S * [A](t) * [B](t)^2 - (K+F) * [B](t)) * dT$$
</p>

<h4 id="diffusion-1">Diffusion</h4>

<p>We can write the following equations for a “two-cell” system :</p>

<ul>
  <li>The amount leaked out of a cell \(C_y\) over duration \(dT\)
is \(out_X(t,C_y) = \tau * [X](t,C_y) * dT\).</li>
  <li>Anything that leaks out of a cell \(C_y\) gets inside the other cell \(C_z\) :
\(in_X(t,C_z) = out_X(t,C_y)\)</li>
  <li>The variation of quantity in a cell \(C_y\) is the difference between the quantity
that leaked in and the quantity that leaked out :
\([X](t+dT,C_y) - [X](t,C_y) = in_X(t,C_y) - out_X(t,C_y)\)
for \(y=1\) and \(y=2\)</li>
</ul>

<p>By rearranging these equations, we get the following update rule for diffusion :</p>

<p>
$$[X](t+dT,C_y) = [X](t,C_y) + \tau_X * [X](t,C_z) * dT - \tau_X * [X](t,C_y)) * dT $$
$$[X](t+dT,C_y) = (1-\tau_X * dT) * [X](t,C_y) + \tau_X * dT * [X](t,C_z)$$
</p>

<p>with \((y,z) = (1,2)\) or \((y,z) = (2,1)\).</p>

<p>This update rule can be generalized from a two-cell system to the 2D grid by
replacing \([X](t,C_z)\) by a (possibly weighted) average of the concentrations
in the neighbor cells.</p>

<h3 id="visual-representation-of-the-system">Visual representation of the system</h3>

<p>Before starting to write code, let’s pick a way to display the state of the
simulation. Since we only need to represent the concentrations of two chemical
species over a 2D system, the full state of the system can be represented, at
any given time, with a picture. Each pixel in this picture is a cell in the
simulated grid, and the red and green channels of each pixel are respectively
proportional to the concentrations of species \(A\) and \(C\) in the corresponding
grid cells.</p>

<h3 id="texture-buffer">Texture buffer</h3>

<p>In order to run the simulation, we will need to store the current state of the
grid (the concentration of each chemical species for each cell) at a location
we are able to read during a later iteration. This is required in order to apply
the update rules, which use the state at time \(t\) to compute a new state at time
\(t+dT\).</p>

<p>To achieve this, we will use a secondary shader that will get rendered to a <em>texture buffer</em>. The
main shader will simply display this texture to the screen, while the secondary
shader will be in charge of actually running the simulation.</p>

<p>To create the secondary shader, click on the “+” in the tab bar of the editor and
select “Buffer A”. This will create a new tab in which you can write another
<code class="language-plaintext highlighter-rouge">mainImage(...)</code> function. We will call it the <em>simulation shader</em>.</p>

<p>Make sure this both shaders have access to Buffer A’s
contents from the previous iteration : map it to <code class="language-plaintext highlighter-rouge">iChannel0</code> in both the
“Image” and “Buffer A” tabs.</p>

<p>The output of the <em>simulation</em> shader from the previous time step will be available
to both shaders as the <em>uniform</em> <code class="language-plaintext highlighter-rouge">iChannel0</code>, from which we can retrieve the
value of a pixel using <code class="language-plaintext highlighter-rouge">texture(iChannel0, vec2(x, y))</code>, where <code class="language-plaintext highlighter-rouge">x</code> and <code class="language-plaintext highlighter-rouge">y</code> are
the coordinates of the pixel we’re interested in. Note that both these coordinates
are floats with values between 0.0 and 1.0, no matter the size in pixels of the texture.</p>

<p>Since the main shader is only responsible for displaying the contents of
<code class="language-plaintext highlighter-rouge">iChannel0</code> to the screen, we can already write the full code for it and focus
on the actual simulation later :</p>

<div class="language-glsl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">mainImage</span><span class="p">(</span> <span class="k">out</span> <span class="kt">vec4</span> <span class="n">fragColor</span><span class="p">,</span> <span class="k">in</span> <span class="kt">vec2</span> <span class="n">fragCoord</span> <span class="p">)</span>
<span class="p">{</span>
    <span class="c1">// Normalized pixel coordinates (from 0 to 1)</span>
    <span class="kt">vec2</span> <span class="n">uv</span> <span class="o">=</span> <span class="n">fragCoord</span><span class="o">/</span><span class="n">iResolution</span><span class="p">.</span><span class="n">xy</span><span class="p">;</span>

    <span class="c1">// Output to screen</span>
    <span class="n">fragColor</span> <span class="o">=</span> <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span><span class="p">)</span> <span class="o">*</span> <span class="kt">vec4</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">5</span><span class="p">,</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">.,</span><span class="mi">1</span><span class="p">.);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>This code simply converts <code class="language-plaintext highlighter-rouge">fragCoord</code> (which contains coordinates in pixel
units) to <code class="language-plaintext highlighter-rouge">uv</code> (coordinates between 0 and 1). It then samples <code class="language-plaintext highlighter-rouge">iChannel0</code> at the
<code class="language-plaintext highlighter-rouge">uv</code> coordinates. Finally, it scales the red channel by <code class="language-plaintext highlighter-rouge">0.5</code> and the green
channel by <code class="language-plaintext highlighter-rouge">2.0</code>, which will make it easier to visually interpret the
simulation.</p>

<p><strong>Update :</strong> I’ve been playing with different ways to map the contents of the
concentrations of reactants to colors. I’m really happy with a color-scheme
based on the <a href="https://en.wikipedia.org/wiki/HSL_and_HSV">Hue-Saturation-Value
(HSV)</a> color representation. It is a
lot more complex and uses some “magic numbers” to make the result look nice.
Since this new color scheme lets us see new details, specifically at the border of
“cells” and around them, I’ve updated the code on Shadertoy to use it. Red and
green still respectively represent “only food” and “only catalyst”, but hues
from cyan to purple represent intermediate mixes of the two reactants.</p>

<h3 id="simulation-shader">Simulation shader</h3>

<p>This shader will repeatedly apply the update rules to every pixel in the
<code class="language-plaintext highlighter-rouge">iChannel0</code> texture buffer, effectively running the simulation. It consists of a
<code class="language-plaintext highlighter-rouge">mainImage(...)</code> function, just like the main shader.</p>

<h4 id="initialization">Initialization</h4>

<p>Let’s start with defining some variables we will need and setting an initial
state for the simulation.</p>

<div class="language-glsl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">mainImage</span><span class="p">(</span> <span class="k">out</span> <span class="kt">vec4</span> <span class="n">fragColor</span><span class="p">,</span> <span class="k">in</span> <span class="kt">vec2</span> <span class="n">fragCoord</span> <span class="p">)</span>
<span class="p">{</span>
    <span class="kt">float</span> <span class="n">dT</span> <span class="o">=</span> <span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">;</span> <span class="c1">//The lower this is, the more stable (but the slower) the simulation is. Weird stuff starts to happen from 3.0</span>
    <span class="kt">vec4</span> <span class="n">TAU</span> <span class="o">=</span> <span class="kt">vec4</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">4</span><span class="p">,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">.,</span> <span class="mi">0</span><span class="p">.);</span>  <span class="c1">// Diffusion rate of components</span>
    <span class="kt">float</span> <span class="n">k1</span> <span class="o">=</span> <span class="mi">1</span><span class="p">.;</span>  <span class="c1">// Speed constant of the main reaction</span>
    <span class="kt">float</span> <span class="n">k2</span> <span class="o">=</span> <span class="mi">0</span><span class="p">.</span><span class="mo">057</span><span class="p">;</span> <span class="c1">// Speed constant of the "kill" reaction</span>
    <span class="kt">float</span> <span class="n">k3</span> <span class="o">=</span> <span class="mi">0</span><span class="p">.</span><span class="mo">01</span><span class="mi">95</span><span class="p">;</span> <span class="c1">// Speed constant of the "feed" reaction</span>
    
    <span class="kt">vec2</span> <span class="n">pixelSize</span> <span class="o">=</span> <span class="mi">1</span><span class="p">.</span> <span class="o">/</span> <span class="n">iResolution</span><span class="p">.</span><span class="n">xy</span><span class="p">;</span>
    <span class="kt">vec2</span> <span class="n">uv</span> <span class="o">=</span> <span class="n">fragCoord</span><span class="p">.</span><span class="n">xy</span> <span class="o">*</span> <span class="n">pixelSize</span><span class="p">;</span>
    
    <span class="kt">vec2</span> <span class="n">h</span> <span class="o">=</span> <span class="kt">vec2</span><span class="p">(</span><span class="n">pixelSize</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="mi">0</span><span class="p">.);</span>
    <span class="kt">vec2</span> <span class="n">v</span> <span class="o">=</span> <span class="kt">vec2</span><span class="p">(</span><span class="mi">0</span><span class="p">.,</span> <span class="n">pixelSize</span><span class="p">.</span><span class="n">y</span><span class="p">);</span>
    
    <span class="k">if</span> <span class="p">(</span><span class="n">iFrame</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Init</span>
        <span class="n">fragColor</span> <span class="o">=</span> <span class="kt">vec4</span><span class="p">(</span><span class="mi">1</span><span class="p">.,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="p">);</span> <span class="c1">// Default initial value</span>
        <span class="k">if</span> <span class="p">(</span> <span class="n">length</span><span class="p">(</span><span class="n">uv</span> <span class="o">-</span> <span class="kt">vec2</span><span class="p">(.</span><span class="mi">5</span><span class="p">,.</span><span class="mi">5</span><span class="p">))</span> <span class="o">&lt;</span> <span class="n">length</span><span class="p">(</span><span class="mi">1</span><span class="p">.</span><span class="o">*</span><span class="n">pixelSize</span><span class="p">)</span> <span class="p">)</span> <span class="p">{</span>  <span class="c1">// If at center of canvas</span>
            <span class="n">fragColor</span><span class="o">+=</span> <span class="kt">vec4</span><span class="p">(</span><span class="mi">0</span><span class="p">.,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">.,</span> <span class="mi">1</span><span class="p">.);</span>  <span class="c1">// Seed-in some catalyst</span>
        <span class="p">}</span>
    <span class="p">}</span>
    <span class="k">else</span> <span class="p">{</span>  <span class="c1">// Simulation</span>
        <span class="c1">// Apply the update rules</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>The first block of definitions are the constants we will use in the update
rules for the reactions and for diffusion :</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">dT</code> is the duration that will be simulated at every step</li>
  <li><code class="language-plaintext highlighter-rouge">TAU</code> (\(\tau\)) is a vector that holds the diffusion rates of \(A\) and \(C\)
in its first and second components.</li>
  <li><code class="language-plaintext highlighter-rouge">k1</code>, <code class="language-plaintext highlighter-rouge">k2</code> and <code class="language-plaintext highlighter-rouge">k3</code> are the speed constants for the 3 chemical reactions that take
place in the system</li>
</ul>

<p>All these values have been cherry picked to produce an interesting result.
Later, we’ll see that you can get results that look very different by tuning
them.</p>

<p>The second block of declarations are useful for converting between coordinates
in pixel space and <em>UV</em> coordinates (floats between 0.0 and 1.0).</p>

<p><code class="language-plaintext highlighter-rouge">h</code> and <code class="language-plaintext highlighter-rouge">v</code> are vectors in UV coordinates that are 1 pixel long, respectively
horizontally and vertically. This will be useful for computing the update rule
for diffusion, which involves retrieving the values from the neighboring pixels.</p>

<p>The conditional block handles the initialization of the simulation by outputting
a fixed state for the first 10 frames. This initial state consists of only \(A\)
everywhere, except for a small radius in the middle where we add a small amount of
the catalyst \(C\).</p>

<p>After these 10 initial frames, all subsequent executions of the
shader will go through the <code class="language-plaintext highlighter-rouge">else</code> block, in which we will implement the actual
update rules.</p>

<h4 id="update-rules-implementation">Update rules implementation</h4>

<p>Here is the full contents of the <code class="language-plaintext highlighter-rouge">else</code> block :</p>

<div class="language-glsl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">vec4</span> <span class="n">col</span> <span class="o">=</span> <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span><span class="p">);</span>

<span class="kt">vec4</span> <span class="n">inboundFlow</span> <span class="o">=</span> <span class="n">TAU</span> <span class="o">/</span> <span class="mi">8</span><span class="p">.</span> <span class="o">*</span> <span class="p">(</span>   <span class="c1">// Algebric inbound diffusion flow</span>
    <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span> <span class="o">+</span> <span class="n">h</span><span class="p">)</span> <span class="o">+</span>  <span class="c1">// Concentrations of neighbor to the right</span>
    <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span> <span class="o">-</span> <span class="n">h</span><span class="p">)</span> <span class="o">+</span>  <span class="c1">// Concentrations of neighbor to the left</span>
    <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span> <span class="o">+</span> <span class="n">v</span><span class="p">)</span> <span class="o">+</span>  <span class="c1">// ....</span>
    <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span> <span class="o">-</span> <span class="n">v</span><span class="p">)</span> <span class="o">+</span>
    <span class="mi">1</span><span class="p">.</span><span class="o">/</span><span class="mi">1</span><span class="p">.</span><span class="mi">41</span><span class="o">*</span><span class="p">(</span>  <span class="c1">// diagonal neighbors are at a distance of sqrt(2) ~= 1.41</span>
        <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span> <span class="o">+</span> <span class="n">h</span> <span class="o">+</span> <span class="n">v</span><span class="p">)</span> <span class="o">+</span>
        <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span> <span class="o">+</span> <span class="n">h</span> <span class="o">-</span> <span class="n">v</span><span class="p">)</span> <span class="o">+</span>
        <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span> <span class="o">-</span> <span class="n">h</span> <span class="o">+</span> <span class="n">v</span><span class="p">)</span> <span class="o">+</span>
        <span class="n">texture</span><span class="p">(</span><span class="n">iChannel0</span><span class="p">,</span> <span class="n">uv</span> <span class="o">-</span> <span class="n">h</span> <span class="o">-</span> <span class="n">v</span><span class="p">)</span>
        <span class="p">)</span> <span class="o">-</span>
    <span class="mi">4</span><span class="p">.</span><span class="o">*</span><span class="p">(</span><span class="mi">1</span><span class="p">.</span><span class="o">+</span><span class="mi">1</span><span class="p">.</span><span class="o">/</span><span class="mi">1</span><span class="p">.</span><span class="mi">41</span><span class="p">)</span><span class="o">*</span><span class="n">col</span>
<span class="p">);</span>

<span class="c1">// Reaction : X + 2X -&gt; 3X</span>
<span class="kt">float</span> <span class="n">reactionSpeed1</span> <span class="o">=</span> <span class="n">k1</span><span class="o">*</span><span class="n">col</span><span class="p">.</span><span class="n">x</span><span class="o">*</span><span class="n">col</span><span class="p">.</span><span class="n">y</span><span class="o">*</span><span class="n">col</span><span class="p">.</span><span class="n">y</span><span class="p">;</span>
<span class="c1">// Concentration variations due to reactions</span>
<span class="kt">vec4</span> <span class="n">dCol</span> <span class="o">=</span> <span class="kt">vec4</span><span class="p">(</span><span class="o">-</span><span class="n">reactionSpeed1</span> <span class="o">+</span> <span class="n">k3</span><span class="o">*</span><span class="p">(</span><span class="mi">1</span><span class="p">.</span><span class="o">-</span><span class="n">col</span><span class="p">.</span><span class="n">x</span><span class="p">),</span> <span class="n">reactionSpeed1</span> <span class="o">-</span> <span class="p">(</span><span class="n">k2</span><span class="o">+</span><span class="n">k3</span><span class="p">)</span><span class="o">*</span><span class="n">col</span><span class="p">.</span><span class="n">y</span><span class="p">,</span> <span class="mi">0</span><span class="p">.,</span> <span class="mi">0</span><span class="p">.);</span>

<span class="n">fragColor</span> <span class="o">=</span> <span class="n">clamp</span><span class="p">(</span><span class="n">col</span><span class="o">+</span><span class="n">dT</span><span class="o">*</span><span class="p">(</span><span class="n">dCol</span><span class="o">+</span><span class="n">inboundFlow</span><span class="p">),</span> <span class="mi">0</span><span class="p">.,</span> <span class="mi">1</span><span class="p">.);</span>
</code></pre></div></div>

<p>This can be broken up into 4 steps :</p>

<ol>
  <li>Retrieve the concentrations at the start of the time step from <code class="language-plaintext highlighter-rouge">iChannel0</code>
and store it inside <code class="language-plaintext highlighter-rouge">col</code></li>
  <li>Compute the variations of concentrations due to diffusion and store the
result inside <code class="language-plaintext highlighter-rouge">inboundFlow</code></li>
  <li>Compute the variations of concentrations due to all chemical reactions and
store the result inside <code class="language-plaintext highlighter-rouge">dCol</code></li>
  <li><code class="language-plaintext highlighter-rouge">dCol</code> and <code class="language-plaintext highlighter-rouge">inboundFlow</code> are scaled by <code class="language-plaintext highlighter-rouge">dT</code> before being added to the
concentrations at the start of the time step, which gives us the
concentrations at the end of the time step. This is then clamped between 0.0
and 1.0 before being used as the output of the shader</li>
</ol>

<p>Every time a frame is rendered, this shader is executed once for every pixel of
the canvas, resulting in a full update of the simulation. Since this shader’s
output is <code class="language-plaintext highlighter-rouge">iChannel0</code>, this means that on each frame (iteration of the
simulation), <code class="language-plaintext highlighter-rouge">col</code> will hold the result from the previous time step.</p>

<h4 id="final-shader">Final shader</h4>

<p>We can now put everything together to obtain <a href="https://www.shadertoy.com/view/lXXcz7">the code that runs the simulation
in this article’s introduction</a>. There
are a few additions to the code I just presented here, which make the shader
interactive :</p>

<ul>
  <li>The <code class="language-plaintext highlighter-rouge">iMouse</code> uniform lets us add some catalyst anywhere by clicking on the simulation :</li>
</ul>

<div class="language-glsl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// The following goes at the beginning of the `else` block</span>
<span class="kt">vec4</span> <span class="n">new</span> <span class="o">=</span> <span class="kt">vec4</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="n">iMouse</span><span class="p">.</span><span class="n">z</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">.</span><span class="mi">5</span> <span class="o">&amp;&amp;</span> <span class="n">length</span><span class="p">((</span><span class="n">fragCoord</span> <span class="o">-</span> <span class="n">iMouse</span><span class="p">.</span><span class="n">xy</span><span class="p">)</span><span class="o">*</span><span class="n">pixelSize</span><span class="p">)</span> <span class="o">&lt;</span> <span class="n">length</span><span class="p">(</span><span class="mi">1</span><span class="p">.</span><span class="o">*</span><span class="n">pixelSize</span><span class="p">)</span> <span class="p">)</span> <span class="p">{</span>
    <span class="c1">// If the mouse button is pressed AND the pixel we're drawing is at the mouse's location</span>
    <span class="n">new</span> <span class="o">=</span> <span class="kt">vec4</span><span class="p">(</span><span class="mi">0</span><span class="p">.,</span> <span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">.,</span> <span class="mi">1</span><span class="p">.);</span>
<span class="p">}</span>

<span class="c1">// Contents of the `else` block from the previous section</span>

<span class="c1">// The final line of the `else` block is now :</span>
<span class="n">fragColor</span> <span class="o">=</span> <span class="n">clamp</span><span class="p">(</span><span class="n">col</span><span class="o">+</span><span class="n">new</span><span class="o">+</span><span class="n">dT</span><span class="o">*</span><span class="p">(</span><span class="n">dCol</span><span class="o">+</span><span class="n">inboundFlow</span><span class="p">),</span> <span class="mi">0</span><span class="p">.,</span> <span class="mi">1</span><span class="p">.);</span> <span class="c1">// Same as before, but we add `new`</span>
</code></pre></div></div>

<ul>
  <li>We map the keyboard state to the uniform <code class="language-plaintext highlighter-rouge">iChannel1</code> in Buffer A and reset the simulation
when the space bar is pressed. The keyboard state is represented as a 2D
texture in which the current state of the spacebar can be read from
the red channel at coordinates <code class="language-plaintext highlighter-rouge">(0.126953125, 0.25)</code> :</li>
</ul>

<div class="language-glsl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Replace the `if(...)` line with :</span>
<span class="kt">bool</span> <span class="n">spacePressed</span> <span class="o">=</span> <span class="n">texture</span><span class="p">(</span><span class="n">iChannel1</span><span class="p">,</span>  <span class="kt">vec2</span><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">126953125</span><span class="p">,</span> <span class="p">.</span><span class="mi">25</span><span class="p">)).</span><span class="n">x</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">.;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">iFrame</span> <span class="o">&lt;</span> <span class="mi">10</span> <span class="o">||</span> <span class="n">spacePressed</span><span class="p">)</span> <span class="p">{</span> 
</code></pre></div></div>

<p>I also moved the simulation’s parameters to the “Common” tab in Shadertoy, with
the goal of making it easier to tune these parameters and observe the wide
range of behaviors that can emerge from this model.</p>

<h2 id="playing-with-the-simulation">Playing with the simulation</h2>

<h3 id="interesting-emergent-behaviors">Interesting emergent behaviors</h3>

<p>Some smart people have already studied the Gray-Scott model in-depth. And
described the behaviors for some interesting parameter values. Most
prominently :</p>

<ul>
  <li><a href="http://mrob.com/pub/comp/xmorphia/index.html">Robert Munafo hosts a methodic and exhaustive exploration of the Gray-Scott model’s
parameter space</a>. His website
also features a gallery of videos for interesting parameter values.</li>
  <li><a href="https://karlsims.com/rd.html">Karl Sims’ tutorial</a> features some visual
explanations.</li>
  <li><a href="https://itp.uni-frankfurt.de/~gros/StudentProjects/Projects_2020/projekt_schulz_kaefer/">Katharina Käfer and Mirjam Schulz’s
page</a>
features an interesting <a href="https://itp.uni-frankfurt.de/~gros/StudentProjects/Projects_2020/projekt_schulz_kaefer/#theory">“Theory”
section</a>
that includes some discussion about the model’s fixed points (conditions over
which concentrations in both chemical species in a cell remain constant over time).</li>
</ul>

<p>Their work has allowed me to quickly try out speed-constant values which produce
interesting results. The video below showcases a few set of well-known parameters.</p>

<iframe width="560" height="315" src="https://www.youtube.com/embed/rFwKSS5C3e8?si=rrxew4JtzG42cgNC" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay;
clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen=""></iframe>

<p>I strongly encourage you to <a href="https://www.shadertoy.com/view/lXXcz7">check it out on Shadertoy</a>.
Experiment with different values for the speed constants <code class="language-plaintext highlighter-rouge">k2</code> and <code class="language-plaintext highlighter-rouge">k3</code> (in the “Common” tab)
and see how adding some catalyst (green species) with your mouse changes the
patterns.</p>

<h3 id="hacking-on-the-code">Hacking on the code</h3>

<p>Beyond simply tuning the values of <code class="language-plaintext highlighter-rouge">k2</code> (\(K\)) and <code class="language-plaintext highlighter-rouge">k3</code> (\(F\)), there are a few easy things to try 
(use the “Fork” button to save your changes to a new shader) :</p>

<ul>
  <li>Change the initial conditions of the simulation and see how they influence the emerging behaviors for some given parameter values. The easiest way to do this is to change the source of <code class="language-plaintext highlighter-rouge">iChannel2</code> in the “Buffer A” tab and change the <code class="language-plaintext highlighter-rouge">channel2Init</code> parameter in the “Common” tab</li>
  <li>Add new ways to interact with the simulation, such as :
    <ul>
      <li>A keyboard toggle for adding/removing the catalyst with the mouse</li>
      <li>A key to switch between the catalyst (green) and the food (red) when using the mouse</li>
      <li>A key to pause the simulation while maintaining the ability to interact with it</li>
      <li>Different keys reset the simulation with a different initial state</li>
    </ul>
  </li>
  <li>Control the scale of the simulation by expanding the diffusion neighborhood</li>
  <li>Use some <a href="https://lygia.xyz/generative">generative noise</a> to continuously add some perturbations to the simulation</li>
  <li>Try to reproduce the last piece of footage from the video (parameter space visualization).
This only requires setting <code class="language-plaintext highlighter-rouge">k2</code> and <code class="language-plaintext highlighter-rouge">k3</code> to be proportional to the x and y coordinates. If you go one step
further, you can even zoom into the most interesting regions of the parameter space.</li>
</ul>

<h3 id="continuous-cellular-automata">Continuous cellular automata</h3>

<p>You may have heard of <a href="https://conwaylife.com/">Conway’s game of life</a> in which a
grid is repeatedly updated according to simple local rules in a binary way (the cells
are either ON or OFF). This is an instance of a <a href="https://en.wikipedia.org/wiki/Cellular_automaton">cellular automaton</a></p>

<p><img src="/media/conway-glider-generator.gif" alt="A glider generator in Conway's game of life" /></p>

<p>Even though Conway’s game of life is much simpler than the reaction-diffusion
model, complex patterns can still emerge from it such as the one in the
animation above. Some people have even built logic gates in Conway’s game of life
and used them to build <a href="https://www.youtube.com/watch?v=QtJ77qsLrpw">Game of life inside Game of
life</a></p>

<p>The Gray-Scott model can be seen as a continuous extension to the discrete version :
instead of each cell’s state being represented with a value from a finite set of
possible values, each cell’s state is now represented as two numbers from a
continuum of possible values.</p>

<p>Other continuous cellular automata include the
<a href="https://chakazul.github.io/lenia.html#Code">Lenia</a> family and <a href="https://sites.google.com/view/flowlenia/">Flow
Lenia</a> which adds constraints
enforcing some conservation of mass in the system. These kinds of models are
actually used by scientific researchers to explore possible conditions for the
emergence of proto-life.</p>

<p>Taking inspiration from the implementation I wrote for the Gray-Scott model, it
should be possible to run a rudimentary version of other continuous cellular
automata.</p>

<p><strong>Update :</strong> You can take a look at <a href="https://slackermanz.com/understanding-multiple-neighborhood-cellular-automata/">Slackermanz’s blog
post</a>
and <a href="https://www.shadertoy.com/user/SlackermanzCA">Shadertoy profile</a> for a
similar shader-based implementation of such continuous cellular automata.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Although this can be considered an unorthodox use of shaders, this has been a
great way to introduce myself to GPU programming. Being an implementation of
concepts I’m already familiar with, I think this was much easier than dealing
with the complex linear algebra involved in 3D rendering.</p>

<p>I hope to explore more systems that exhibit emergence in future posts, as I find
this field really fascinating.</p>]]></content><author><name>Pierre Couy</name></author><category term="Simulations" /><category term="shader" /><category term="simulation" /><category term="emergence" /><category term="scientific programming" /><summary type="html"><![CDATA[Use the parallel processing power of your GPU to simulate a simple chemical system that exhibits emergent behaviors]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://pierre-couy.dev/media/gray-scott-grid.png" /><media:content medium="image" url="https://pierre-couy.dev/media/gray-scott-grid.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Increase privacy by using nginx as a caching proxy in front of a map tile server</title><link href="https://pierre-couy.dev/server-admin/2024/08/proxying-a-map-tile-server-for-increased-privacy.html" rel="alternate" type="text/html" title="Increase privacy by using nginx as a caching proxy in front of a map tile server" /><published>2024-08-30T00:00:00+00:00</published><updated>2024-08-30T00:00:00+00:00</updated><id>https://pierre-couy.dev/server-admin/2024/08/proxying-a-map-tile-server-for-increased-privacy</id><content type="html" xml:base="https://pierre-couy.dev/server-admin/2024/08/proxying-a-map-tile-server-for-increased-privacy.html"><![CDATA[<p>If you are self-hosting any service, chances are that you care about
increasing your privacy by minimizing your reliance on third-party services. If
this is the case, you may be bothered when an application you are hosting relies
on such third-parties. This can be the case when some features are too resource
intensive for personal servers.</p>

<p>One example of this is map tile servers, which are relied upon for map features
in a variety of software, such as <a href="https://immich.app/">Immich</a> (awesome Google Photos replacement !).
Such tile servers host a whole world map at several zoom levels, and provide
clients with map fragments (tiles) for the requested coordinates and zoom
level.
Unfortunately, it is not easy to host such a tile server : the <a href="https://protomaps.com/">easiest solution
I could find</a> still requires more than 100GB of disk
space to serve a full world map. On the other hand, using a third party for this
makes clients send a bunch of requests to them. These requests will give the
third-party details about any location viewed on the map. It will also generally
include other informations, such as the URL you’re viewing the map from and
clients IP addresses.</p>

<p>This article will show how to build a caching reverse proxy in order to mitigate these
privacy concerns while avoiding the need to host more than 100GB of map data.
As a concrete application, the caching proxy will then be used as a tile
provider for an Immich instance.</p>

<h2 id="why-do-this">Why do this</h2>

<p>Hosting a caching proxy between the clients and the tile provider can bring
several general benefits :</p>

<ul>
  <li>Limit the amount of personally identifiable information (PII) sent to the tile
provider by using the Immich instance’s IP address and stripping the
<code class="language-plaintext highlighter-rouge">Referer</code> header</li>
  <li>Limit the frequency at which PII is sent to the tile provider by caching tiles
that have already been loaded through the proxy</li>
  <li>The decreased load on the upstream tile provider makes it reasonable to use Open
Street Map’s tile server as the upstream provider</li>
  <li>The upstream provider will not be able to differentiate between several users
of the same proxy</li>
</ul>

<p>In addition, there are a few Immich specific advantages (which may apply to
other similar software) :</p>

<ul>
  <li>If you do not need the map displaying your photos to be perfectly up to date,
you can set an arbitrarily long caching duration</li>
  <li>Most use cases for the map will frequently zoom on the same areas, which makes
it a good fit for a cache</li>
</ul>

<h2 id="tutorial">Tutorial</h2>

<p>This guide will use <a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html">Nginx proxy module</a>
to build a caching proxy in front of Open Street Map’s tileserver and to serve a custom
<code class="language-plaintext highlighter-rouge">style.json</code> for the maps.</p>

<p>This works if you already proxy your services behind an Nginx instance.
It is probably possible to achieve similar results with other reverse proxies,
but this would obviously need to be adapted.</p>

<p>While this guide is directed towards Immich users, the nginx configuration can
be easily used with other applications. As long as it provides a way to switch
tile providers, you should be able to use your proxy with it.</p>

<p>If you get stuck trying to follow this guide, feel free to ask anything <a href="https://github.com/pcouy/pcouy.github.io/discussions/1">in the
support thread on GitHub</a>.</p>

<h3 id="caching-proxy">Caching proxy</h3>

<p>Inside Nginx’s <code class="language-plaintext highlighter-rouge">http</code> config block (usually in <code class="language-plaintext highlighter-rouge">/etc/nginx/nginx.conf</code>), create
a cache zone (a directory that will hold cached responses from OSM) :</p>

<div class="language-nginx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">http</span> <span class="p">{</span>
    <span class="c1"># You should not need to edit existing lines in the http block, only add the line below</span>
    <span class="kn">proxy_cache_path</span> <span class="n">/var/cache/nginx/osm</span> <span class="s">levels=1:2</span> <span class="s">keys_zone=osm:100m</span> <span class="s">max_size=5g</span> <span class="s">inactive=180d</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>You may need to manually create the <code class="language-plaintext highlighter-rouge">/var/cache/nginx/osm</code> directory and set its
owner to Nginx’s user (typically <code class="language-plaintext highlighter-rouge">www-data</code> on Debian based distros).</p>

<p>Customize the <code class="language-plaintext highlighter-rouge">max_size</code> parameter to change the maximum amount of cached data
you want to store on your server. The <code class="language-plaintext highlighter-rouge">inactive</code> parameter will cause Nginx to
discard cached data that’s not been accessed in this duration (180d ~ 6months).</p>

<p>Then, inside the <code class="language-plaintext highlighter-rouge">server</code> block that serves your Immich instance, create a new
<code class="language-plaintext highlighter-rouge">location</code> block :</p>

<div class="language-nginx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">server</span> <span class="p">{</span>
    <span class="kn">listen</span> <span class="mi">443</span> <span class="s">ssl</span><span class="p">;</span>
    <span class="kn">server_name</span> <span class="s">immich.your-domain.tld</span><span class="p">;</span>

    <span class="c1"># You should not need to change your existing config, only add the location block below</span>

    <span class="kn">location</span> <span class="n">/map_proxy/</span> <span class="p">{</span>
        <span class="kn">proxy_pass</span> <span class="s">https://tile.openstreetmap.org/</span><span class="p">;</span>
        <span class="kn">proxy_cache</span> <span class="s">osm</span><span class="p">;</span>
        <span class="kn">proxy_cache_valid</span> <span class="s">180d</span><span class="p">;</span>
        <span class="kn">proxy_ignore_headers</span> <span class="s">Cache-Control</span> <span class="s">Expires</span><span class="p">;</span>
        <span class="kn">proxy_ssl_server_name</span> <span class="no">on</span><span class="p">;</span>
        <span class="kn">proxy_ssl_name</span> <span class="s">tile.openstreetmap.org</span><span class="p">;</span>
        <span class="kn">proxy_set_header</span> <span class="s">Host</span> <span class="s">tile.openstreetmap.org</span><span class="p">;</span>
        <span class="kn">proxy_set_header</span> <span class="s">User-Agent</span> <span class="s">"Nginx</span> <span class="s">Caching</span> <span class="s">Tile</span> <span class="s">Proxy</span> <span class="s">for</span> <span class="s">self-hosters"</span><span class="p">;</span>
        <span class="kn">proxy_set_header</span> <span class="s">Cookie</span> <span class="s">""</span><span class="p">;</span>
        <span class="kn">proxy_set_header</span> <span class="s">Referer</span> <span class="s">""</span><span class="p">;</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Reload Nginx (<code class="language-plaintext highlighter-rouge">sudo systemctl reload nginx</code>). Confirm this works by visiting
<code class="language-plaintext highlighter-rouge">https://immich.your-domain.tld/map_proxy/0/0/0.png</code>, which should now return a
world map PNG (the one from https://tile.openstreetmap.org/0/0/0.png )</p>

<p>This config ignores cache control headers from OSM and sets its own cache
validity duration (<code class="language-plaintext highlighter-rouge">proxy_cache_valid</code> parameter). After the specified duration,
the proxy will re-fetch the tiles. 6 months seem reasonable to me for the use
case, and it can probably be set to a few years without it causing issues.</p>

<p>Besides being lighter on OSM’s servers, the caching proxy will improve privacy
by only requesting tiles from upstream when loaded for the first time. This
config also strips cookies and referrer before forwarding the queries to OSM, as
well as set a user agent for the proxy following <a href="https://operations.osmfoundation.org/policies/tiles/">OSM foundation’s
guidelines</a> (according to
these guidelines, you should add a contact information to this user agent)</p>

<p>This can probably be made to work on a different domain than the one serving
your Immich instance, but this will require tweaking CORS headers.</p>

<h3 id="custom-stylejson">Custom <code class="language-plaintext highlighter-rouge">style.json</code></h3>

<p>The following map style can be used to replace Immich’s default tile provider
with your caching proxy :</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="p">,</span><span class="w">
  </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Immich Map"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"sources"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"immich-map"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"raster"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"tileSize"</span><span class="p">:</span><span class="w"> </span><span class="mi">256</span><span class="p">,</span><span class="w">
      </span><span class="nl">"tiles"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"https://immich.your-domain.tld/map_proxy/{z}/{x}/{y}.png"</span><span class="p">]</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">},</span><span class="w">
  </span><span class="nl">"sprite"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://maputnik.github.io/osm-liberty/sprites/osm-liberty"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"glyphs"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://fonts.openmaptiles.org/{fontstack}/{range}.pbf"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"layers"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
    </span><span class="p">{</span><span class="w">
      </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"raster-tiles"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"raster"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"immich-map"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"minzoom"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
      </span><span class="nl">"maxzoom"</span><span class="p">:</span><span class="w"> </span><span class="mi">22</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">],</span><span class="w">
  </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"immich-map-dark"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p>Replace <code class="language-plaintext highlighter-rouge">immich.your-domain.tld</code> with your actual Immich domain, and remember
the absolute path you save this at on your server.</p>

<h3 id="one-last-update-to-nginxs-config">One last update to nginx’s config</h3>

<p>Since Immich currently does not provide a way to manually edit <code class="language-plaintext highlighter-rouge">style.json</code>, we
need to serve it from http(s). Add one more <code class="language-plaintext highlighter-rouge">location</code> block below the previous
one :</p>

<div class="language-nginx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">location</span> <span class="n">/map_style.json</span> <span class="p">{</span>
    <span class="kn">alias</span> <span class="n">/srv/immich/mapstyle.json</span><span class="p">;</span> <span class="c1"># This needs to be the location where you saved the file from the previous step</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Replace the <code class="language-plaintext highlighter-rouge">alias</code> parameter with the location where you saved the json
map style. After reloading nginx, your json style will be available at
<code class="language-plaintext highlighter-rouge">https://immich.your-domain.tld/map_style.json</code>. You can now use this URL to
your style as both the light and dark themes in your instance’s settings.</p>

<p>You can now <a href="https://immich.app/docs/guides/custom-map-styles">set up your Immich instance to use your new JSON map style</a></p>]]></content><author><name>Pierre Couy</name></author><category term="Server admin" /><category term="linux" /><category term="nginx" /><category term="tutorial" /><category term="privacy" /><category term="selfhosting" /><summary type="html"><![CDATA[A tutorial featuring two examples showing how you can increase your privacy using nginx to proxy third-party services.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://pierre-couy.dev/media/banner-immich-nginx-caching.png" /><media:content medium="image" url="https://pierre-couy.dev/media/banner-immich-nginx-caching.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Using a Raspberry Pi to add a second HDMI port to a laptop</title><link href="https://pierre-couy.dev/tinkering/2023/03/turning-rpi-into-external-monitor-driver.html" rel="alternate" type="text/html" title="Using a Raspberry Pi to add a second HDMI port to a laptop" /><published>2023-03-11T00:00:00+00:00</published><updated>2023-03-11T00:00:00+00:00</updated><id>https://pierre-couy.dev/tinkering/2023/03/turning-rpi-into-external-monitor-driver</id><content type="html" xml:base="https://pierre-couy.dev/tinkering/2023/03/turning-rpi-into-external-monitor-driver.html"><![CDATA[<p>Recently, I purchased a new laptop. I was really focused on spending the least amount of money and had not noticed that the laptop I chose was missing an essential feature : it did not have Display Port over USB C. Not being able to use my second external monitor on this new laptop felt like a huge downgrade from my previous one (which was able to output to both its HDMI and VGA ports simultaneously).</p>

<p>This is the story of how I managed to overcome this limitation by rolling my own virtual screen streaming solution using a Raspberry Pi. I tried to write it in a way you can follow along if you want to reproduce it. If you are just looking to get it up and running as quick as possible, you can check out <a href="https://github.com/pcouy/rpi-eth-display">the GitHub repository containing configuration files and installation scripts</a> (Work In Progress)</p>

<p>You will find a short video showcasing the result at the end of this article.</p>

<h2 id="existing-solutions-and-limitations-of-old-raspberry-pi-models">Existing solutions and limitations of old Raspberry Pi models</h2>

<p>I quickly hooked a Raspberry Pi to the external monitor and tried to find a turnkey solution that would allow me to stream a virtual screen to the Pi via an Ethernet cable. I looked into using VNC, Steam Remote Play, and some dedicated VNC wrappers I found on GitHub.</p>

<p>Since I was not willing to spend more money on my setup, I used a Raspberry Pi 3 which was sitting unused in one of my drawers. This little beasts support hardware-accelerated video decoding, including h264. However, as we’ll see later, my specific requirements made it harder to work with GPU video decoders. I had to compromise between picture quality, latency and framerate, and could never reach a balance I felt satisfied with : the slow LAN port and CPU could not handle my requirements.</p>

<p>I also did not like the fact that most of these solutions depended on running a full desktop session on the Pi, which I wanted to avoid in order to save its thin resources.</p>

<h2 id="goals">Goals</h2>

<p>Since I intended to use this daily, and I could not see myself using anything I had tried, I decided to go for my own solution. I had a clear goal in mind : after setting it up, it should feel as much as using a regular external monitor as possible ; while still being able to run on outdated hardware.</p>

<p>My main requirements were the following :</p>

<ul>
  <li>The latency should not be noticeable when scrolling or moving the mouse</li>
  <li>The picture quality should be high enough to read small text</li>
  <li>Since I planned to mainly use it for static text content, I decided to go easy on myself by setting a low target of 10 FPS.</li>
  <li>If the receiving end of the stream ever gets behind, it should catch-up to live as quick as possible</li>
  <li>Use <a href="https://en.wikipedia.org/wiki/Direct_Rendering_Manager">Direct Rendering Manager</a> to display the stream on the Pi instead of depending on a X server.</li>
  <li>I looked into remote-play tools and VNC because they seemed like easy to use low-latency solutions. However, I was not interested with streaming inputs back from the Pi to the laptop.</li>
</ul>

<p>As I was using a Raspberry Pi 3, I had to consider its limitations :</p>

<ul>
  <li>Due to slow CPU, use a low-overhead protocol and fast to decode encoding</li>
  <li>Due to slow network, use a low-bitrate encoding</li>
  <li>No hardware accelerated h264 decoding (this is not a limitation of the Pi 3 per se, but after experimentation <a href="https://www.reddit.com/r/programming/comments/11r9osx/comment/jc9s8zp/">using the <code class="language-plaintext highlighter-rouge">v4l2m2m</code> codecs negated all my optimizations regarding latency</a>)</li>
</ul>

<p>Since I was already going to roll my own solution, I also listed some non essential features I would enjoy having, including :</p>

<ul>
  <li>Having a DHCP server on the Raspberry Pi so that I would not have to bother myself with IP settings</li>
  <li>Automatically running the necessary software on the Pi at boot so I never have to hook a keyboard or SSH into it for regular use</li>
  <li>Having the laptop automatically start streaming to the Pi when I enable a given virtual monitor with <code class="language-plaintext highlighter-rouge">xrandr</code> (or one of its GUI wrapper such as <code class="language-plaintext highlighter-rouge">arandr</code>)</li>
  <li>Automatically turning the pi-controlled monitor on and off as if it were a regular monitor hooked to a regular HDMI port</li>
</ul>

<h2 id="making-it-happen">Making it happen</h2>

<p>I knew the hardest part was going to fine-tune the video pipeline between the laptop and the Pi. I wanted to tackle this first and only spend time on other features when I was sure it was worth it.</p>

<p>I chose to encode and send the stream using <a href="https://ffmpeg.org/"><code class="language-plaintext highlighter-rouge">ffmpeg</code></a> on my laptop (which is known to be the Swiss-army knife of audio and video manipulation). It takes care of screen-grabbing, video encoding, encapsulation and networking and provides fine-grained controls over all steps. Its numerous options can often feel overwhelming, but digging the docs have never let me down.</p>

<p>For the receiving end, I considered several <code class="language-plaintext highlighter-rouge">ffmpeg</code>-compatible video players with Direct Rendering Manager support, including <code class="language-plaintext highlighter-rouge">mpv</code>, <code class="language-plaintext highlighter-rouge">vlc</code>, and <code class="language-plaintext highlighter-rouge">ffplay</code> (more on that topic later).</p>

<h3 id="raspberry-pi-initial-setup">Raspberry Pi initial setup</h3>

<p>I started with a fresh Raspberry Pi OS install, which I flashed on my SD card using the usual commands :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pierre@laptop:~ $</span><span class="w"> </span>lsblk <span class="nt">-f</span> <span class="c"># Identify SD card block device</span>
<span class="gp">pierre@laptop:~ $</span><span class="w"> </span><span class="nb">sudo dd </span><span class="k">if</span><span class="o">=</span>2022-09-22-raspios-bullseye-arm64-lite.img <span class="nv">of</span><span class="o">=</span>/dev/sd[SD card letter]</code></pre></figure>

<p>I booted the Pi a first time with the screen and a keyboard attached. This lets Raspberry Pi OS resize the partition to fit the SD card. After connecting the Pi to my home WiFi and enabling SSH using <a href="https://www.raspberrypi.com/documentation/computers/configuration.html"><code class="language-plaintext highlighter-rouge">raspi-config</code></a>, I unplugged the keyboard from the Pi and SSH’ed into it.</p>

<p>I installed the required software to quickly start experimenting with the stream settings :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span><span class="nb">sudo </span>apt-get update <span class="o">&amp;&amp;</span> <span class="nb">sudo </span>apt-get <span class="nb">install </span>mpv ffmpeg</code></pre></figure>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pierre@laptop:~ $</span><span class="w"> </span><span class="nb">sudo </span>apt-get update <span class="o">&amp;&amp;</span> <span class="nb">sudo </span>apt-get <span class="nb">install </span>ffmpeg</code></pre></figure>

<p>While waiting for the players to install, I found an Ethernet cable to use between the Pi and the laptop. To my surprise, both computers seemed to be able to talk to each other without me doing anything, so I started tinkering with <code class="language-plaintext highlighter-rouge">ffmpeg</code> parameters. I don’t remember the details, but the connection ended up not being stable enough. It was necessary to install and configure a DHCP server on the Raspberry Pi in order to comfortably experiment.</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>udhcpd
<span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span>sudoedit /etc/udhcpd.conf</code></pre></figure>

<p>This will install <a href="https://manpages.ubuntu.com/manpages/bionic/man5/udhcpd.conf.5.html"><code class="language-plaintext highlighter-rouge">udhcpd</code></a> and open its configuration file with root privileges using the editor set in your <code class="language-plaintext highlighter-rouge">EDITOR</code> shell variable (<code class="language-plaintext highlighter-rouge">nano</code> by default on Raspberry Pi OS). I used the following configuration file :</p>

<figure class="highlight"><pre><code class="language-conf" data-lang="conf"><span class="c"># Only one lease for the Pi itself, and one for the laptop
</span><span class="n">start</span> <span class="m">10</span>.<span class="m">0</span>.<span class="m">0</span>.<span class="m">1</span>
<span class="n">end</span> <span class="m">10</span>.<span class="m">0</span>.<span class="m">0</span>.<span class="m">2</span>

<span class="c"># udhcpd will use eth0
</span><span class="n">interface</span> <span class="n">eth0</span>

<span class="c"># Various options
</span><span class="n">option</span> <span class="n">subnet</span> <span class="m">255</span>.<span class="m">255</span>.<span class="m">255</span>.<span class="m">252</span>
<span class="n">option</span> <span class="n">domain</span> <span class="n">hdmi</span>
<span class="n">option</span> <span class="n">lease</span>  <span class="m">60</span>  <span class="c"># One minute lease
</span>
<span class="c"># The Pi itself will always be 10.0.0.1
</span><span class="n">static_lease</span> [<span class="n">PI</span> <span class="n">MAC</span> <span class="n">ADDRESS</span>] <span class="m">10</span>.<span class="m">0</span>.<span class="m">0</span>.<span class="m">1</span></code></pre></figure>

<p>You will need to replace <code class="language-plaintext highlighter-rouge">[PI MAC ADDRESS]</code> with the actual MAC address of your hardware, which you can find by running <code class="language-plaintext highlighter-rouge">ip a</code> on the Pi (<code class="language-plaintext highlighter-rouge">link/ether</code> field).</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pi@rapsberrypi:~ $</span><span class="w"> </span><span class="nb">sudo </span>systemctl <span class="nb">enable </span>udhcpd
<span class="gp">pi@rapsberrypi:~ $</span><span class="w"> </span><span class="nb">sudo </span>systemctl restart udhcpd</code></pre></figure>

<p>The first command above will launch the DHCP server on boot, and the second one will launch it immediately. Rebooting the Pi may help both computers pick up on their new network configurations. From now on, the Raspberry Pi will be reachable from the laptop using <code class="language-plaintext highlighter-rouge">10.0.0.1</code> as long as the Ethernet cable is plugged to both. The laptop will use the IP <code class="language-plaintext highlighter-rouge">10.0.0.2</code>.</p>

<h3 id="starting-an-unoptimized-stream">Starting an unoptimized stream</h3>

<p>With this initial setup done, I was able to quickly iterate over commands for sending and receiving the stream. This was not a straightforward process and while I did not keep records of every attempt, I’ll do my best to tell the interesting discoveries I made along the way. I will also detail every option in the commands presented below.</p>

<p>On the Raspberry Pi, the goal was to launch a media player that would listen on the network waiting for the laptop to send it a stream, and display it using DRM with the lowest possible latency. I first tried using <a href="https://mpv.io/"><code class="language-plaintext highlighter-rouge">mpv</code></a> because of its support for GPU decoding.</p>

<p>Since both ends of the stream were connected over a single wire with no realistic opportunity for interception and I wanted to save resources on the Pi, encryption was not necessary. My requirements for lowest possible latency led my to try streaming over plain UDP. Long story short, my experiments with UDP did not go so well : one skipped packet and the whole screen would turn to garbage (or worse, the player would crash). I then switched to TCP, which proved to offer low-enough latency while not suffering from the same issue.</p>

<p>Let’s start with the most basic command that does that, without bothering with optimization for now :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span>mpv <span class="nt">--hwdec</span><span class="o">=</span>drm <span class="s2">"tcp://10.0.0.1:1234?listen"</span></code></pre></figure>

<p>This command makes <code class="language-plaintext highlighter-rouge">mpv</code> listen on interface <code class="language-plaintext highlighter-rouge">10.0.0.1</code>, TCP port <code class="language-plaintext highlighter-rouge">1234</code> and will display the received stream using DRM.</p>

<p>On the sending side, I started with a simple command to test the stream :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pierre@laptop:~ $</span><span class="w"> </span>ffmpeg <span class="nt">-video_size</span> 1920x1080 <span class="nt">-framerate</span> 5 <span class="nt">-f</span> x11grab <span class="nt">-i</span> :0.0+0x0 <span class="nt">-f</span> mpegts tcp://10.0.0.1:1234</code></pre></figure>

<p>From <code class="language-plaintext highlighter-rouge">man ffmpeg</code>, the syntax is :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="go">ffmpeg [global_options] {[input_file_options] -i input_url} ... {[output_file_options] output_url}</span></code></pre></figure>

<p>Let’s detail the arguments used here :</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">-video_size 1920x1080</code> indicates the size of the region to grab.</li>
  <li><code class="language-plaintext highlighter-rouge">-framerate 5</code> only grabs 5 frames per second. This is below our requirement but this allows somewhat smooth testing of the setup before optimization.</li>
  <li><a href="https://ffmpeg.org/ffmpeg-devices.html#x11grab"><code class="language-plaintext highlighter-rouge">-f x11grab</code></a> : used as an input file option, <code class="language-plaintext highlighter-rouge">-f</code> specifies the input device. <code class="language-plaintext highlighter-rouge">x11grab</code> is used for screen grabbing.</li>
  <li><code class="language-plaintext highlighter-rouge">-i :0.0+0x0</code> : <code class="language-plaintext highlighter-rouge">-i</code> is usually used for specifying input file. When used with the X11 video input device, specifies where to grab from in the syntax : <code class="language-plaintext highlighter-rouge">[hostname]:display_number.screen_number[+x_offset,y_offset]</code></li>
  <li><a href="https://ffmpeg.org/ffmpeg-formats.html#mpegts"><code class="language-plaintext highlighter-rouge">-f mpegts</code></a> : used as an output file option, <code class="language-plaintext highlighter-rouge">-f</code> specifies the output container (also called file format or muxer). <code class="language-plaintext highlighter-rouge">mpegts</code> designates MPEG-2 transport stream.</li>
  <li><code class="language-plaintext highlighter-rouge">tcp://10.0.0.1:1234</code> is the URL to send the stream to (the <code class="language-plaintext highlighter-rouge">mpv</code> listener running on the Pi)</li>
</ul>

<p>This did not meet any of my performance and quality requirements, but provided me with a starting point I could optimize from.</p>

<h3 id="optimizing-the-receiving-end-of-the-stream">Optimizing the receiving end of the stream</h3>

<p>I then tried two optimization strategies on the receiving side, which involved a lot of googling and a bunch of not-so-well documented <code class="language-plaintext highlighter-rouge">mpv</code> options :</p>

<ul>
  <li>Speeding up decoding using hardware acceleration</li>
  <li>Jumping to the latest available frame when decoding fell behind</li>
</ul>

<p>I came up with the following <code class="language-plaintext highlighter-rouge">mpv</code> command (which I will not detail) before trying another player :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span>mpv <span class="nt">-vo</span><span class="o">=</span>gpu <span class="nt">--gpu-context</span><span class="o">=</span>drm <span class="nt">--input-cursor</span><span class="o">=</span>no <span class="nt">--input-vo-keyboard</span><span class="o">=</span>no <span class="nt">--input-default-bindings</span><span class="o">=</span>no <span class="nt">--hwdec</span><span class="o">=</span>drm <span class="nt">--untimed</span> <span class="nt">--no-cache</span> <span class="nt">--profile</span><span class="o">=</span>low-latency <span class="nt">--opengl-glfinish</span><span class="o">=</span><span class="nb">yes</span> <span class="nt">--opengl-swapinterval</span><span class="o">=</span>0 <span class="nt">--gpu-hwdec-interop</span><span class="o">=</span>drmprime-drm <span class="nt">--drm-draw-plane</span><span class="o">=</span>overlay <span class="nt">--drm-drmprime-video-plane</span><span class="o">=</span>primary <span class="nt">--framedrop</span><span class="o">=</span>no <span class="nt">--speed</span><span class="o">=</span>1.01 <span class="nt">--video-latency-hacks</span><span class="o">=</span><span class="nb">yes</span> <span class="nt">--opengl-glfinish</span><span class="o">=</span><span class="nb">yes</span> <span class="nt">--opengl-swapinterval</span><span class="o">=</span>0 tcp://10.0.0.1:1234<span class="se">\?</span>listen</code></pre></figure>

<p>While this achieved the best latency I could reach using <code class="language-plaintext highlighter-rouge">mpv</code> and the basic <code class="language-plaintext highlighter-rouge">ffmpeg</code> command above, I felt this was too complicated. Some other resources I found online were using <a href="https://ffmpeg.org/ffplay.html"><code class="language-plaintext highlighter-rouge">ffplay</code></a> on the receiving end so I gave it a try. This proved to be a much simpler path, and I achieved comparable results using the following command :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span>ffplay <span class="nt">-autoexit</span> <span class="nt">-flags</span> low_delay <span class="nt">-framedrop</span> <span class="nt">-strict</span> experimental <span class="nt">-vf</span> <span class="nv">setpts</span><span class="o">=</span>0 <span class="nt">-tcp_nodelay</span> 1 <span class="s2">"tcp://10.0.0.1:1234</span><span class="se">\?</span><span class="s2">listen"</span></code></pre></figure>

<p>Most of these optimizations came from <a href="https://stackoverflow.com/questions/16658873/how-to-minimize-the-delay-in-a-live-streaming-with-ffmpeg">this StackOverflow post about minimizing delay in a live stream</a>. Let’s detail the meaning of the options I used :</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">-autoexit</code> makes <code class="language-plaintext highlighter-rouge">ffplay</code> exit when the stream ends</li>
  <li><a href="https://ffmpeg.org/ffplay-all.html#Codec-Options"><code class="language-plaintext highlighter-rouge">-flags low_delay</code></a> seemed like an obvious choice, even if the documentation is not clear about what it does</li>
  <li><a href="https://ffmpeg.org/ffplay-all.html#Advanced-options"><code class="language-plaintext highlighter-rouge">-framedrop</code></a> “Drop video frames if video is out of sync”</li>
  <li><a href="https://ffmpeg.org/ffplay-all.html#Codec-Options"><code class="language-plaintext highlighter-rouge">-strict experimental</code></a> enables “unfinished/work in progress/not well tested” stuff. This proved to be useful. Note : the documentation mentions this option not being suitable for decoding untrusted input. You should probably remove it if you plan on plugging untrusted computers on your Raspberry Pi’s LAN port.</li>
  <li><a href="https://ffmpeg.org/ffplay-all.html#setpts_002c-asetpts"><code class="language-plaintext highlighter-rouge">-vf setpts=0</code></a> : <code class="language-plaintext highlighter-rouge">-vf</code> is used to specify video filters. The <code class="language-plaintext highlighter-rouge">setpts</code> filter changes the <em>Presentation TimeStamp</em> of video frames. <code class="language-plaintext highlighter-rouge">setpts=0</code> is used to make all frames display as soon as possible</li>
  <li><code class="language-plaintext highlighter-rouge">-tcp_nodelay 1</code> enables the <a href="https://www.extrahop.com/company/blog/2016/tcp-nodelay-nagle-quickack-best-practices/">TCP nodelay flag</a>. I’m not sure this one really had any impact, but it made sense to include it and did not hurt performances.</li>
</ul>

<p>The stream sent by the basic <code class="language-plaintext highlighter-rouge">ffmpeg</code> command gets displayed on the Pi monitor with a delay of approximately 1 second using <code class="language-plaintext highlighter-rouge">ffplay</code>. This is too high, and the quality is too low for small text, but we are very close to the final command I’m still running on the Pi.</p>

<p>Let’s make sure the OS prioritizes the <code class="language-plaintext highlighter-rouge">ffplay</code> process using the <code class="language-plaintext highlighter-rouge">nice</code> and <code class="language-plaintext highlighter-rouge">ionice</code> commands :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span><span class="nb">sudo nice</span> <span class="nt">-n</span> <span class="nt">-20</span> ionice <span class="nt">-c</span> 1 <span class="nt">-n</span> 0 ffplay <span class="nt">-autoexit</span> <span class="nt">-flags</span> low_delay <span class="nt">-framedrop</span> <span class="nt">-strict</span> experimental <span class="nt">-vf</span> <span class="nv">setpts</span><span class="o">=</span>0 <span class="nt">-tcp_nodelay</span> 1 <span class="s2">"tcp://10.0.0.1:1234</span><span class="se">\?</span><span class="s2">listen"</span></code></pre></figure>

<h3 id="supervising-ffplay">Supervising <code class="language-plaintext highlighter-rouge">ffplay</code></h3>

<p>Since the player automatically detects, decodes and demuxes the input codec and muxer, I could experiment with the sending side without changing the command run on the Pi. However, I still had to switch between terminals in order to manually restart <code class="language-plaintext highlighter-rouge">ffplay</code> between each try. This pushed me to take care of a non-essential feature before going on.</p>

<p>I used <a href="http://supervisord.org/"><code class="language-plaintext highlighter-rouge">supervisor</code></a> to manage the media player process. The choice was motivated by its ease of use over creating <code class="language-plaintext highlighter-rouge">systemd</code> services.</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>supervisor
<span class="gp">pi@raspberrypi:~ $</span><span class="w"> </span>sudoedit /etc/supervisor/conf.d/pimonitor.conf</code></pre></figure>

<p>This will install <code class="language-plaintext highlighter-rouge">supervisor</code> and open a configuration file for editing. I used the following content :</p>

<figure class="highlight"><pre><code class="language-conf" data-lang="conf">[<span class="n">program</span>:<span class="n">ffplay</span>]
<span class="n">command</span>=<span class="n">nice</span> -<span class="n">n</span> -<span class="m">20</span> <span class="n">ionice</span> -<span class="n">c</span> <span class="m">1</span> -<span class="n">n</span> <span class="m">0</span> <span class="n">ffplay</span> -<span class="n">autoexit</span> -<span class="n">flags</span> <span class="n">low_delay</span> -<span class="n">framedrop</span> -<span class="n">strict</span> <span class="n">experimental</span> -<span class="n">vf</span> <span class="n">setpts</span>=<span class="m">0</span> -<span class="n">tcp_nodelay</span> <span class="m">1</span> <span class="s2">"tcp://10.0.0.1:1234\?listen"</span>
<span class="n">autorestart</span>=<span class="n">true</span>
<span class="n">stdout_logfile</span>=/<span class="n">dev</span>/<span class="n">null</span>
<span class="n">stderr_logfile</span>=/<span class="n">dev</span>/<span class="n">null</span></code></pre></figure>

<p>The <code class="language-plaintext highlighter-rouge">autorestart</code> option makes a new instance of <code class="language-plaintext highlighter-rouge">ffplay</code> listen and wait for a new stream when the previous one exits. I used <code class="language-plaintext highlighter-rouge">/dev/null</code> for logfiles to prevent <code class="language-plaintext highlighter-rouge">ffplay</code>’s verbose output from filling my small SD card with log files.</p>

<p>After starting the <code class="language-plaintext highlighter-rouge">supervisor</code> daemon with <code class="language-plaintext highlighter-rouge">sudo systemctl enable supervisor</code> and <code class="language-plaintext highlighter-rouge">sudo systemctl restart supervisor</code>, I could try <code class="language-plaintext highlighter-rouge">ffmpeg</code> option combinations much quicker.</p>

<h3 id="fine-tuning-the-encoder-process">Fine-tuning the encoder process</h3>

<p>The first thing I did was increase the framerate to 30 FPS, and I was really surprised to find out this helped a lot with latency. The encoder would still occasionally fall behind, which caused latency spikes, but the with that simple change it suddenly started to feel like I was on the right track.</p>

<p>I then tried switching from the default <code class="language-plaintext highlighter-rouge">mpeg2video</code> to the more modern <code class="language-plaintext highlighter-rouge">mpeg4</code> which did not lead to any improvement in itself, but provided more options. Switching the muxer from <code class="language-plaintext highlighter-rouge">mpegts</code> to <code class="language-plaintext highlighter-rouge">nut</code> led to more noticeable improvements regarding delay. While quality was still too low, it started to feel responsive enough to meet the latency requirement.</p>

<p>I then managed to increase the quality to my standards by using encoder options to target a higher bit-rate (<code class="language-plaintext highlighter-rouge">-b:v 40M -maxrate 50M -bufsize 200M</code>). However, the Raspberry Pi became overloaded and started to drop a couple of frames a few times per seconds. This led to an unpleasant experience, with the mouse movements and scrolling not feeling smooth. What surprised me the most was seeing frames being dropped even when displaying a still screen.</p>

<h4 id="hunting-down-the-framedrops">Hunting down the framedrops</h4>

<p>At this point, I was back to square one, trying to find the balance between picture quality and smoothness. One key difference, however, was that this time I was working with tools I was somewhat familiar with, and provided lots of options. After trying a few things that did not work, I noticed a few things :</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">ffmpeg</code> was sending a stream with a bitrate of several Mbit/s for a still screen.</li>
  <li>Framedrops from <code class="language-plaintext highlighter-rouge">ffplay</code> seemed to happen at a very stable rate.</li>
  <li>The Raspberry Pi did not seem to be limited by its CPU.</li>
</ul>

<p>This hinted to me that the problem came from the network, so I launched a network capture using <code class="language-plaintext highlighter-rouge">tcpdump</code> :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pierre@laptop:~ $</span><span class="w"> </span><span class="nb">sudo </span>tcpdump <span class="nt">-i</span> eth0 <span class="nt">-c</span> 2000 <span class="nt">-w</span> diag_remote_screen.pcapng <span class="s2">"port 1234"</span>
<span class="gp">pierre@laptop:~ $</span><span class="w"> </span>tcpdump <span class="nt">-r</span> diag_remote_screen.pcapng | <span class="nb">awk</span> <span class="s1">'{ print $1 " " $8 " " $9 " " $NF }'</span> | less</code></pre></figure>

<p>This captures 2000 packets of the stream between <code class="language-plaintext highlighter-rouge">ffmpeg</code> running on the laptop and <code class="language-plaintext highlighter-rouge">ffplay</code> running on the Pi. The second command is used to examine the captured packets, but you can also open the <code class="language-plaintext highlighter-rouge">.pcapng</code> file with Wireshark or other similar tools.</p>

<p>The command above shows :</p>

<ul>
  <li>The time at which the packet was captured</li>
  <li>The TCP sequence number for packets from the laptop to the Pi and their acknowledgments</li>
  <li>The size of packets</li>
</ul>

<p>Here is a sample of its output :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="go">14:13:36.879965 seq 79239:81556, 2317
14:13:36.881709 ack 81556, 0
14:13:36.916838 seq 81556:83849, 2293
14:13:36.918185 ack 83849, 0
14:13:36.943326 seq 83849:85014, 1165
14:13:36.944438 ack 85014, 0
14:13:36.981337 seq 85014:87613, 2599
14:13:36.982724 ack 87613, 0
14:13:37.014469 seq 87613:88769, 1156
14:13:37.015752 ack 88769, 0
14:13:37.054639 seq 88769:90701, 1932
14:13:37.055851 ack 90701, 0
14:13:37.077741 seq 90701:91858, 1157
14:13:37.079045 ack 91858, 0
14:13:37.121258 seq 91858:107786, 15928
14:13:37.121301 seq 107786:123714, 15928
14:13:37.121324 seq 123714:124626, 912
14:13:37.121360 seq 124626:140554, 15928
14:13:37.121374 seq 140554:156482, 15928
14:13:37.121386 seq 156482:172410, 15928
14:13:37.121391 seq 172410:188338, 15928
14:13:37.121403 seq 188338:204266, 15928
14:13:37.121410 seq 204266:220194, 15928
14:13:37.121421 seq 220194:236122, 15928
14:13:37.121426 seq 236122:252050, 15928
14:13:37.121438 seq 252050:267978, 15928
14:13:37.122535 seq 267978:283906, 15928
14:13:37.122567 ack 94754, 0
14:13:37.122567 ack 97650, 0
14:13:37.122567 ack 100546, 0
14:13:37.122585 seq 283906:299834, 15928
14:13:37.123237 ack 103442, 0
14:13:37.123237 ack 106338, 0
14:13:37.123238 ack 109234, 0
14:13:37.123255 seq 299834:315762, 15928
14:13:37.123891 seq 315762:331690, 15928
14:13:37.123916 seq 331690:347618, 15928
14:13:37.123926 ack 112130, 0
    [LOTS OF SUCCESSIVE ACKs]
14:13:37.135636 ack 254946, 0
14:13:37.136070 seq 347618:363546, 15928
14:13:37.136273 ack 257842, 0
14:13:37.136273 ack 260738, 0
14:13:37.136273 ack 263634, 0
14:13:37.136989 ack 266530, 0
14:13:37.136989 ack 269426, 0
14:13:37.136989 ack 272322, 0
    [REPEAT 25x THE ABOVE PATTERN OF A 15928 BYTES TCP PACKET FOLLOWED BY A FEW ACKs]
14:13:37.168585 seq 745818:761746, 15928
14:13:37.169275 ack 645906, 0
14:13:37.169275 ack 648802, 0
14:13:37.169275 ack 651698, 0
14:13:37.169857 seq 761746:769413, 7667
14:13:37.170274 ack 654594, 0
    [LOTS OF SUCCESSIVE ACKs]
14:13:37.179345 ack 769413, 0
14:13:37.184011 seq 769413:770863, 1450
14:13:37.185333 ack 770863, 0
14:13:37.214388 seq 770863:772194, 1331
14:13:37.215822 ack 772194, 0
14:13:37.241472 seq 772194:774010, 1816
14:13:37.243176 ack 774010, 0</span></code></pre></figure>

<p>At first, we see the laptop sends a packet that weights a couple kB approximately every 0.033s, which matches our framerate of 30fps. The Pi sends the acknowledgments for each of these packets before the next one comes in. At <code class="language-plaintext highlighter-rouge">14:13:37.121258</code>, <code class="language-plaintext highlighter-rouge">ffmpeg</code> starts sending a lot of 16kB packets to the Pi and the acknowledgment numbers start falling behind. When the Pi gets too far behind, <code class="language-plaintext highlighter-rouge">ffmpeg</code> waits for ACKs to catch-up a little before sending more data (TCP sequence numbers <code class="language-plaintext highlighter-rouge">283906-769413</code>). This burst of data from the laptop stops at <code class="language-plaintext highlighter-rouge">14:13:37.169857</code> (TCP seq num <code class="language-plaintext highlighter-rouge">769413</code>) and the Pi TCP stack finally catches up at <code class="language-plaintext highlighter-rouge">14:13:37.179345</code> (TCP ack <code class="language-plaintext highlighter-rouge">769413</code>). This is <code class="language-plaintext highlighter-rouge">0.58s</code> (almost 2 frames) after the laptop began sending this data. This whole thing happened precisely every 12 frames and explained the details I noticed earlier about the framedrops.</p>

<p>The MPEG codec compresses videos by only saving a few frames in full, which are called keyframes. All other frames are derived from the previous frame which is associated with a description of the differences between consecutive frames. Data bursts occur every-time <code class="language-plaintext highlighter-rouge">ffmpeg</code> sends a keyframe, which is set by default to happen every 12 frame (~ 3 times/sec).</p>

<p>Increasing the “group of picture” <a href="https://ffmpeg.org/ffmpeg-codecs.html#Codec-Options">codec option</a> from 12 to 100 (~ once every 3 seconds) had the expected effect : framedrops were only happening once every 3 seconds, which I could live with.</p>

<p>At this point I had the following command :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pierre@laptop:~ $</span><span class="w"> </span>ffmpeg <span class="nt">-video_size</span> 1920x1080 <span class="nt">-framerate</span> 30 <span class="se">\</span>
<span class="go">    -f x11grab -i :0.0+0x0 \
    -b:v 40M -maxrate 50M -bufsize 200M \
    -vcodec mpeg4 -g 100 -f nut \
    "tcp://10.0.0.1:1234"</span></code></pre></figure>

<p>Even though I was satisfied with what I managed to get, I kept tinkering with options. At one point, it became difficult to tell what actually improved the experience and what could be attributed to some kind of placebo effect. Anyway, here is the final command I came up with :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pierre@laptop:~ $</span><span class="w"> </span>ffmpeg <span class="nt">-video_size</span> 1920x1080 <span class="nt">-r</span> 30 <span class="nt">-framerate</span> 30 <span class="nt">-f</span> x11grab <span class="nt">-i</span> :0.0+0x0 <span class="se">\</span>
<span class="go">    -b:v 40M -maxrate 50M -bufsize 200M \
    -field_order tt -fflags nobuffer -threads 1 \
    -vcodec mpeg4 -g 100 -r 30 -bf 0 -mbd bits -flags +aic+mv4+low_delay \
    -thread_type slice -slices 1 -level 32 -strict experimental -f_strict experimental \
    -syncpoints none -f nut "tcp://10.0.0.1:1234"</span></code></pre></figure>

<h3 id="extending-the-laptop-display">Extending the laptop display</h3>

<p>For this task, my goal was to configure the X server on my laptop so that it could output to a virtual monitor I could then screen-grab and stream to the Raspberry Pi.
To accomplish this, I closely followed what <a href="https://github.com/dianariyanto/virtual-display-linux"><code class="language-plaintext highlighter-rouge">virtual-display-linux</code></a> does and I copied the <a href="https://github.com/dianariyanto/virtual-display-linux/blob/master/20-intel.conf">provided configuration file for intel GPU</a>. After rebooting, I could indeed see two monitors called <code class="language-plaintext highlighter-rouge">VIRTUAL1</code> and <code class="language-plaintext highlighter-rouge">VIRTUAL2</code> in my <code class="language-plaintext highlighter-rouge">xrandr</code> output.</p>

<p>Using the accepted answer from <a href="https://unix.stackexchange.com/questions/227876/how-to-set-custom-resolution-using-xrandr-when-the-resolution-is-not-available-i">this StackOverflow thread</a> I created the mode for my external monitor resolution and associated it with the first virtual display :</p>

<figure class="highlight"><pre><code class="language-console" data-lang="console"><span class="gp">pierre@laptop:~ $</span><span class="w"> </span>gtf 1920 1200 30 <span class="c"># gtf {W} {H} {FPS}</span>
<span class="gp">#</span><span class="w"> </span>Use the Modeline from the output of the above <span class="nb">command </span><span class="k">in </span>the <span class="nb">command </span>below
<span class="gp">pierre@laptop:~ $</span><span class="w"> </span>xrandr <span class="nt">--newmode</span> <span class="s2">"1920x1200_30.00"</span>  89.67  1920 1992 2184 2448  1200 1201 1204 1221  <span class="nt">-HSync</span> +Vsync
<span class="gp">pierre@laptop:~ $</span><span class="w"> </span>xrandr <span class="nt">--addmode</span> VIRTUAL1 <span class="s2">"1920x1200_30.00"</span></code></pre></figure>

<p>Note that I used a resolution of 1920x1200 because this is the resolution of the monitor I’m using. If you are following along, you will need to change this to fit your actual screen resolution.</p>

<p>After enabling the virtual monitor using <code class="language-plaintext highlighter-rouge">arandr</code> (a graphical frontend for <code class="language-plaintext highlighter-rouge">xrandr</code>), I modified the <code class="language-plaintext highlighter-rouge">-video_size</code> and <code class="language-plaintext highlighter-rouge">-i</code> options in my <code class="language-plaintext highlighter-rouge">ffmpeg</code> command to grab the virtual display. This worked as intended and it effectively extended my laptop’s display to the Pi-driven monitor.</p>

<h3 id="wrapping-xrandr">Wrapping <code class="language-plaintext highlighter-rouge">xrandr</code></h3>

<p>At this point, my solution was meeting all my primary requirements. I was able to set everything up so it really felt like using a regular monitor. However, I still had to run a bunch of commands by hand on the laptop. How nice would it be to enable the virtual display just like a regular one, and have the <code class="language-plaintext highlighter-rouge">ffmpeg</code> command run automatically with the right options ?</p>

<p>The solution I came up with feels a bit hacky : I wrote a wrapper script for <code class="language-plaintext highlighter-rouge">xrandr</code>.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/bash</span>

<span class="c"># Enable job control</span>
<span class="nb">set</span> <span class="nt">-m</span>

<span class="c"># Extract arguments between `--output VIRTUAL1` and the next occurrence of `--output`</span>
<span class="nv">V_ARGS</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> | <span class="nb">grep</span> <span class="s2">"VIRTUAL1"</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/.*--output VIRTUAL1 //'</span> <span class="nt">-e</span> <span class="s1">'s/ \?--output.*//'</span><span class="si">)</span>

<span class="c"># Run the real xrandr</span>
<span class="c"># (using full path YOU MAY NEED TO UPDATE THIS DEPENDING ON YOUR DISTRO)</span>
/usr/bin/xrandr <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>

<span class="c"># If there were no args related to VIRTUAL1, exit with the same exit code as `xrandr`</span>
<span class="nv">EXITCODE</span><span class="o">=</span><span class="nv">$?</span>
<span class="k">if</span> <span class="o">[</span> <span class="si">$(</span><span class="nb">echo</span> <span class="nv">$V_ARGS</span> | <span class="nb">wc</span> <span class="nt">-w</span><span class="si">)</span> <span class="nt">-eq</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span><span class="nb">exit</span> <span class="nv">$EXITCODE</span>
<span class="k">fi</span>

<span class="c"># Kill the previous ffmpeg process if it exists</span>
<span class="nb">kill</span> <span class="si">$(</span><span class="nb">cat</span> /tmp/remote_screen_ffmpeg.pid<span class="si">)</span>
<span class="nv">KILLEDFFMPEG</span><span class="o">=</span><span class="nv">$?</span>
<span class="nb">rm</span> /tmp/remote_screen_ffmpeg.pid

<span class="c"># If the arguments for the display contain `--off`</span>
<span class="k">if</span> <span class="o">[</span> <span class="si">$(</span><span class="nb">echo</span> <span class="nv">$V_ARGS</span> | <span class="nb">grep</span> <span class="nt">-e</span> <span class="s2">"--off"</span> | <span class="nb">wc</span> <span class="nt">-l</span><span class="si">)</span> <span class="nt">-ge</span> 1 <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span><span class="nb">echo</span> <span class="s2">"Screen off"</span> <span class="o">&gt;&gt;</span> ~/testxrandr <span class="c"># For debugging</span>
<span class="k">else</span>
    <span class="c"># Extract the arguments for the display we're interested in</span>
    <span class="nv">MODE</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="nv">$V_ARGS</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/.*--mode \([^ ]*\).*/\1/'</span><span class="si">)</span>
    <span class="nv">POS</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="nv">$V_ARGS</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/.*--pos \([^ ]*\).*/\1/'</span><span class="si">)</span>
    <span class="nv">ROTATE</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="nv">$V_ARGS</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/.*--rotate \([^ ]*\).*/\1/'</span><span class="si">)</span>

    <span class="c"># If the display is rotated, invert width and height in $MODE</span>
    <span class="k">if</span> <span class="o">[[</span> <span class="nv">$ROTATE</span> <span class="o">==</span> <span class="s2">"left"</span> <span class="o">]]</span> <span class="o">||</span> <span class="o">[[</span> <span class="nv">$ROTATE</span> <span class="o">==</span> <span class="s2">"right"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
        </span><span class="nv">MODE</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="nv">$MODE</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/\([0-9]*\)x\([0-9]*\)/\2x\1/'</span><span class="si">)</span>
    <span class="k">fi</span>

    <span class="c"># $VFARG will be used later in an ffmpeg option</span>
    <span class="k">case</span> <span class="nv">$ROTATE</span> <span class="k">in
        </span>normal<span class="p">)</span>
            <span class="nv">VFARG</span><span class="o">=</span><span class="s2">"null"</span>
            <span class="p">;;</span>
        left<span class="p">)</span>
            <span class="nv">VFARG</span><span class="o">=</span><span class="s2">"transpose=2"</span>
            <span class="p">;;</span>
        right<span class="p">)</span>
            <span class="nv">VFARG</span><span class="o">=</span><span class="s2">"transpose=1"</span>
            <span class="p">;;</span>
        inverted<span class="p">)</span>
            <span class="nv">VFARG</span><span class="o">=</span><span class="s2">"transpose=2,transpose=2"</span>
            <span class="p">;;</span>
        <span class="k">*</span><span class="p">)</span>
            <span class="nv">VFARG</span><span class="o">=</span><span class="s2">"null"</span>
            <span class="p">;;</span>
    <span class="k">esac</span>

    <span class="c"># If there was a previously running ffmpeg process which we killed,</span>
    <span class="c"># wait 5 seconds for the supervisor daemon on the Pi to restart ffplay</span>
    <span class="k">if</span> <span class="o">[</span> <span class="nv">$KILLEDFFMPEG</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
        </span><span class="nb">sleep </span>5
    <span class="k">fi</span>

    <span class="c"># ffmpeg command, the magic happens here</span>
    taskset <span class="nt">-c</span> 0 ffmpeg <span class="nt">-nostdin</span> <span class="se">\</span>
        <span class="nt">-video_size</span> <span class="nv">$MODE</span> <span class="nt">-r</span> 30 <span class="nt">-framerate</span> 30 <span class="nt">-f</span> x11grab <span class="nt">-i</span> :0.0+<span class="nv">$POS</span> <span class="se">\</span>
        <span class="nt">-b</span>:v 40M <span class="nt">-maxrate</span> 50M <span class="nt">-minrate</span> 1K <span class="nt">-bufsize</span> 200M <span class="se">\</span>
        <span class="nt">-field_order</span> tt <span class="nt">-fflags</span> nobuffer <span class="nt">-threads</span> 1 <span class="se">\</span>
        <span class="nt">-vcodec</span> mpeg4 <span class="nt">-g</span> 100 <span class="nt">-r</span> 30 <span class="nt">-bf</span> 0 <span class="se">\</span>
        <span class="nt">-mbd</span> bits <span class="nt">-me_method</span> full <span class="nt">-flags</span> +aic+mv4+low_delay <span class="nt">-me_method</span> full <span class="se">\</span>
        <span class="nt">-thread_type</span> slice <span class="nt">-slices</span> 1 <span class="nt">-level</span> 32 <span class="se">\</span>
        <span class="nt">-strict</span> experimental <span class="nt">-f_strict</span> experimental <span class="nt">-syncpoints</span> none <span class="se">\</span>
        <span class="nt">-vf</span> <span class="s2">"</span><span class="nv">$VFARG</span><span class="s2">"</span> <span class="nt">-f</span> nut <span class="nt">-tcp_nodelay</span> 1 <span class="se">\</span>
        <span class="s2">"tcp://10.0.0.1:1234?tcp_nodelay=1"</span> <span class="o">&gt;</span>/dev/null 2&gt;&amp;1 &amp;

    <span class="c"># Save the ffmpeg pid to a file which we'll read on next invocation</span>
    <span class="nv">FFMPEGPID</span><span class="o">=</span><span class="nv">$!</span>
    <span class="nb">disown</span> <span class="nv">$FFMPEGPID</span>
    <span class="nb">echo</span> <span class="nv">$FFMPEGPID</span> <span class="o">&gt;</span> /tmp/remote_screen_ffmpeg.pid
<span class="k">fi</span>

<span class="c"># Return the same exit code as xrandr did</span>
<span class="nb">exit</span> <span class="nv">$EXITCODE</span></code></pre></figure>

<p>You can recognize the <code class="language-plaintext highlighter-rouge">ffmpeg</code> command from earlier. There are however a few different things :</p>

<ul>
  <li>The <code class="language-plaintext highlighter-rouge">-video_size</code> and <code class="language-plaintext highlighter-rouge">-i</code> options are determined from the <code class="language-plaintext highlighter-rouge">xrandr</code> invocation</li>
  <li>Depending on the screen orientation, we use a <a href="https://ffmpeg.org/ffmpeg-filters.html#transpose-1">video filter</a> to rotate the stream</li>
  <li><code class="language-plaintext highlighter-rouge">ffmpeg</code> is invoked through <a href="https://manpages.ubuntu.com/manpages/trusty/fr/man1/taskset.1.html"><code class="language-plaintext highlighter-rouge">taskset</code></a></li>
</ul>

<p>I saved this script as <code class="language-plaintext highlighter-rouge">~/.local/bin/xrandr</code>. For this to work, you need to have your <code class="language-plaintext highlighter-rouge">~/.local/bin</code> directory in your path, with a higher priority than system-wide directories. This is achieved by adding the following line in your <code class="language-plaintext highlighter-rouge">~/.bashrc</code> (or whatever rc file your shell uses) :</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.local/bin:</span><span class="nv">$PATH</span><span class="s2">"</span></code></pre></figure>

<p>This wrapper script is run every time I run a <code class="language-plaintext highlighter-rouge">xrandr</code> command, including from GUI frontends such as <code class="language-plaintext highlighter-rouge">arandr</code>. It manages the <code class="language-plaintext highlighter-rouge">ffmpeg</code> process and starts the stream whenever the <code class="language-plaintext highlighter-rouge">VIRTUAL1</code> display is enabled. It even manages screen orientation, which was essential to me since I actually use this monitor in portrait orientation.</p>

<h3 id="managing-power">Managing power</h3>

<p>After writing the wrapper script, I was really happy with the result. I even got the pleasant surprise of not having to handle resuming the stream after the laptop wakes up from sleep. Since <code class="language-plaintext highlighter-rouge">ffmpeg</code> was not exiting on sleep, <code class="language-plaintext highlighter-rouge">ffplay</code> silently waited for the laptop to start sending data again. There was one thing bothering me though : I still had to manually power the monitor on and off when leaving my desk.</p>

<p>I googled for how to turn the HDMI port of the Raspberry Pi on and off, and quickly found out about the <a href="https://elinux.org/RPI_vcgencmd_usage"><code class="language-plaintext highlighter-rouge">vcgencmd</code></a> command and its <code class="language-plaintext highlighter-rouge">display_power</code> subcommand. Unfortunately, every command I tried seemed to have no effect on the Raspberry Pi 3. It took me a few days to <a href="https://forum.magicmirror.builders/topic/16865/mmm-remotecontrol-or-vcgencmd-issue">find a fix</a> : by editing the <code class="language-plaintext highlighter-rouge">/boot/config.txt</code> to replace <code class="language-plaintext highlighter-rouge">dtoverlay=vc4-kms-v3d</code> with <code class="language-plaintext highlighter-rouge">dtoverlay=vc4-fkms-v3d</code> and rebooting the Pi, it worked. It seems like the <code class="language-plaintext highlighter-rouge">kms</code> driver has a bug on the Raspberry Pi 3. Fortunately, switching VideoCore drivers did not impact the stream decoding performance. With that issue fixed, I was able to turn the screen on and off from an SSH session.</p>

<p>In order to run the <code class="language-plaintext highlighter-rouge">vcgencmd</code> commands at the right time, I once again went the hacky way and came up with a short script (featuring a dirty infinite loop) :</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/bash</span>

<span class="k">while </span><span class="nb">true</span><span class="p">;</span> <span class="k">do
	if</span> <span class="o">[</span> <span class="si">$(</span><span class="nb">sudo timeout </span>2 tcpdump <span class="nt">-i</span> eth0 <span class="s2">"port 1234"</span> | <span class="nb">wc</span> <span class="nt">-l</span><span class="si">)</span> <span class="nt">-gt</span> 1 <span class="o">]</span><span class="p">;</span> <span class="k">then
		</span>vcgencmd display_power 1 2
	<span class="k">else
		</span>vcgencmd display_power 0 2
	<span class="k">fi
done</span></code></pre></figure>

<p>The loop does the following :</p>

<ul>
  <li>Run <code class="language-plaintext highlighter-rouge">tcpdump</code> for two seconds and count the number of packets received on port 1234 during this time</li>
  <li>If there was at least one packet received during the last 2 seconds, turn the display on</li>
  <li>If no packets were received during the last 2 seconds, turn the display off</li>
  <li>Repeat</li>
</ul>

<p>I saved the script on the Pi as <code class="language-plaintext highlighter-rouge">/home/pi/check_screen_input.sh</code> and edited the <code class="language-plaintext highlighter-rouge">supervisor</code> configuration file :</p>

<figure class="highlight"><pre><code class="language-conf" data-lang="conf">[<span class="n">program</span>:<span class="n">power_mgmt</span>]
<span class="n">command</span>=/<span class="n">home</span>/<span class="n">pi</span>/<span class="n">check_screen_input</span>.<span class="n">sh</span>
<span class="n">autorestart</span>=<span class="n">true</span></code></pre></figure>

<p>I then restarted the <code class="language-plaintext highlighter-rouge">supervisor</code> daemon, which had the effect of stopping the stream. The monitor went back to the Pi tty and after a short moment, turned off. I then disabled and re-enabled the <code class="language-plaintext highlighter-rouge">VIRTUAL1</code> display on my laptop, and the magic happened : the monitor woke up from sleep and extended the laptop’s display.</p>

<h2 id="improvements-and-last-thoughts">Improvements and last thoughts</h2>

<p>I finally reached a solution I could use in my day-to-day life, with only small quirks I don’t mind dealing with. Here’s a video showcasing the setup I’m using daily :</p>

<iframe width="560" height="315" src="https://www.youtube.com/embed/Q_opC1bHSuY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>

<p>I still have to manually create the new mode and add it to the virtual display after every reboot. It would be really nice to have the Pi detect the resolution of the monitor and use it to automatically configure the virtual display on the laptop. However, since I’m of the kind who rarely reboots their computers and I already spent quite some time on this project, I moved on from it without taking care of this part.</p>

<p>The main defect is that I sometimes get visible encoding/decoding glitches that fix themselves on the next keyframe. I don’t know what causes them. If you have leads on this, please open an issue in the GitHub repository.</p>

<p>I made a <a href="https://github.com/pcouy/rpi-eth-display">GitHub repository that features all needed configuration files and scripts, as well as untested installation scripts</a>. The part that runs on the Raspberry Pi seems like a good opportunity to learn how to make a <code class="language-plaintext highlighter-rouge">.deb</code> package, so I may look into it in the future. If there is interest around this project, I may get motivated to make the process more streamlined and beginner-friendly.</p>

<p>Overall, I am really satisfied with what I managed to come up with. While using it, I even noticed I was able to watch videos without the audio-video delay being noticeable. With this solution available, and considering the money it saved me, I may knowingly purchase a laptop that lacks a second video output when I need to replace this one.</p>

<h2 id="updates">Updates</h2>

<h3 id="displaylink">DisplayLink</h3>

<p>Some readers have mentioned that this project is very similar to <a href="https://en.wikipedia.org/wiki/DisplayLink">DisplayLink</a>. I don’t remember coming across this when I did the research for this project. I think this is because the naming makes it ambiguous that this is not the same thing as DP over USB, and I may have dismissed results mentioning it at the time.</p>

<p>After looking more into it, it is indeed really similar to what I did : it requires installing software on the host computer, and uses an active adapter. One key difference though is that the software you must install to use DisplayLink is proprietary, while this project only uses open source parts.</p>

<h3 id="gud">GUD</h3>

<p>Some other readers have mentioned <a href="https://github.com/notro/gud">GUD, which does the same thing I did except it uses USB</a> and looks a lot cleaner on the host side by using a kernel module. I did not really look into the Raspberry Pi side of this project, but I’m making a note to come back to it later.</p>

<h3 id="socket-activated-services">Socket activated services</h3>

<p>If I ever get to turning the Pi-side of the project into a deb package, I will probably make good use of <a href="https://news.ycombinator.com/item?id=35172740">this suggestion to use <code class="language-plaintext highlighter-rouge">systemd</code> socket activated services</a> as a replacement for using <code class="language-plaintext highlighter-rouge">supervisord</code>.</p>]]></content><author><name>Pierre Couy</name></author><category term="Tinkering" /><category term="raspberry-pi" /><category term="ffmpeg" /><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[A step-by-step tutorial you can follow along, featuring ffmpeg, xrandr, tcpdump and a dhcp server running on a Raspberry Pi]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://pierre-couy.dev/media/banner-rpi-eth-monitor.jpg" /><media:content medium="image" url="https://pierre-couy.dev/media/banner-rpi-eth-monitor.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>