Zola2026-03-26T00:00:00+00:00https://blog.graysonhead.net/atom.xmlReliable Delivery Over Horrible Networks2026-03-26T00:00:00+00:002026-03-26T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/reliable-delivery/<p>TCP is a cornerstone of the internet for good reason: it works very well in most situations. One of the reasons it is so effective is because of how it handles dealing with congestion. Congestion control algorithms are like a bouncer at a packed venue. When things are backed up inside, nobody new gets in. When there's room, the line starts moving again.</p>
<p>And as far as TCP is concerned, the best signal for congestion is lost segments <sup class="footnote-reference"><a href="#1">1</a></sup> (segments being the individual unit of data that gets wrapped in an ethernet frame <sup class="footnote-reference"><a href="#2">2</a></sup>). But loss isn't always congestion. Sometimes loss is switching cell towers. Sometimes loss is because you are right on the edge of the effective range of a radio link. Sometimes loss is adversarial interference.</p>
<p>At a certain percentage of loss, you won't be able to deliver any data via TCP at all.</p>
<p>So let's say you are building a network stack for unmanned vehicles that might operate in environments with adversarial jamming. How can you be sure you can deliver a critical message to them in the case of an emergency?</p>
<p>As we will soon discover, it depends.</p>
<h2 id="latency-sucks">Latency Sucks</h2>
<p>The two prime guarantees of TCP are:</p>
<ul>
<li>Reliable Delivery (your byte stream will make it to the other side)</li>
<li>Ordered (your byte stream will arrive in the same order you sent it)</li>
</ul>
<p>And solving these two guarantees efficiently drives much of the design of TCP. And the key to understanding this (and often, to troubleshooting TCP issues) are the "windows" that regulate the flow of data between nodes.</p>
<p>The reason this matters for us: those windows are a feedback system, and feedback systems are sensitive to latency. On a lossy, high-latency link, that sensitivity becomes TCP's biggest weakness.</p>
<p>In an ideal world, the sender would send data as fast as it possibly can, the network would dutifully transport all this data at exactly the rate the sender is sending it, and then the receiver will receive and process the data exactly as fast as the sender sends it. However, in practice, this almost never happens. What if the receiver is an overloaded webserver that is processing data from a million other clients? What if the network is congested? At some point, the sender must slow down. Windows regulate the amount of data that is allowed to be in-flight between clients at any time. They represent the gas pedal that is pressed when all is well and the data is flowing as fast as possible, as well as the brake that gets pushed if the receiver stalls, or the network quality degrades.</p>
<p>The first window is the Send window. This is maintained by the sender. The purpose of this window is to hold the Bandwidth Delay Product (BDP) of the connection. You can figure out what the BDP is with some simple math. Simply multiply the rate of transmission in bytes by the latency in seconds: </p>
<p><code>( BDP = bandwidth * rtt)</code>. </p>
<p>For example, if your application can saturate a 10Gbps NIC and you are sending data to a host 10ms from you, your BDP is <code>10 Gigabits/s * .01</code>. Which works out to around 12.5 Megabytes. In theory, if all you were doing was sending data from one machine to one client, you could set your send buffer <sup class="footnote-reference"><a href="#3">3</a></sup> to this exact value and never need to adjust it. But in practice, this window (as well as all the others) are subject to algorithms that constantly increase and decrease their size to accommodate changing conditions.</p>
<p>So when we send data, data gets put into this buffer. But why don't we forget it? Why does it still stick around after sending? That is because we need to know the other side received it before it can be discarded. We need delivery to be reliable. Once the other side sends us an ACK for that specific segment, then we can drop that specific data from the send buffer and free up room to send more data, but until this happens, the sending side must retain it for possible retransmission. So in addition to being a throttling mechanism, the send buffer can also be a cache for retransmission. To put it simply, the buffer derived from the send window is what allows us to guarantee that the data is both reliably delivered, and ordered.</p>
<p>Next, we have the Receive window. This window regulates how quickly the host on the receiving side of the connection is actually consuming data. Each ack sent by a receiver will contain a <code>rwnd</code> value that indicates how much room is left in the window, and the Sender should not send additional data to the Receiver if it exceeds this value (even if the Sender's send buffer is not full). This allows the receiving host to add backpressure to the system to slow down the propagation of data through a network if it cannot keep up with the rate at which data is being sent.</p>
<p>As an aside, this value is extremely valuable when troubleshooting issues with live applications, you can see it here in wireshark:
<img src="../../images/reliable_delivery/tcp_window.png" alt="Where to find the rwnd in wireshark" /></p>
<p>And finally, we have the congestion window. The congestion window's job is to provide backpressure when the network itself starts introducing problems. In practice, it does the same job as the receive window, but for different reasons. It is adjusted by the operating system based on the measured quality of the connection.</p>
<p>These three windows decide how much data you can send over a TCP connection at any given moment. For instance, if:</p>
<ul>
<li><code>cwnd</code> = 64KiB</li>
<li><code>rwnd</code> = 48KiB</li>
<li>bytes in flight = 30KiB</li>
</ul>
<p>Then you can send <code>min(64, 48) - 30 = 18KiB</code> right now.</p>
<p>So, now that we understand some of the basics, why does latency make this worse?</p>
<p>Remember that on most systems, these window sizes are auto-tuned by the system. In fact, the backpressure and congestion control systems of TCP depend on their size being dynamic. The SEND -> ACK process allows each side of the conversation to discover things about the state of the sender, receiver, and connection between them. But, the quickest they can possibly discover this information is one round trip time <em>after</em> those changes occur. It's a feedback system. And the higher the RTT, the slower the feedback, for both positive (link is underutilized) and negative (high packet loss conditions exist) conditions.</p>
<p>Most algorithms governing this process of optimizing the congestion window additively increase, and multiplicative decrease the cwnd. This means that recovering from a previous loss event is especially punishing, because it takes longer to get back up to the maximum value than it does to go from max -> 0. This exacerbates how long it takes to recover from lossy conditions on high latency links. <em>Especially</em> once you are past the slow start threshold (depicted on this chart as a yellow line) <sup class="footnote-reference"><a href="#4">4</a></sup></p>
<img src="../../images/reliable_delivery/ssh_latency_cwnd.png" alt="Graph showing rate of cwnd increase for different link types" />
<p>Which is all to say, a lossy link is bad for TCP. A high latency lossy link is absolutely terrible.</p>
<p>For instance, let's look at a 1 second 100% loss event on a 20ms network:</p>
<img src="../../images/reliable_delivery/tcp_outage.png" alt="Graph showing recovery after an outage" />
<p>1 second of loss is plenty long enough for the congestion window to clamp down to 0, delivery is completely halted. And then we spend a bit more than a second recovering to full bandwidth.</p>
<p>Now let's look at what this looks like when we go from 20ms to 200ms RTT:</p>
<img src="../../images/reliable_delivery/tcp_outage_geo_single.png" alt="Graph showing the impact of a single outage on a high latency connection" />
<p>We've recreated the same issue, the <code>cwnd</code> clamps to 0, but now recovery takes far longer: almost 3 seconds to get back to full throughput.</p>
<p>Now, let's imagine we have multiple loss events:</p>
<img src="../../images/reliable_delivery/tcp_outage_geo.png" alt="Graph showing impact of multiple outages on a high latency connection" />
<p>As you can see, small bursts of 100% loss can effectively cripple TCP with large latencies. We spend a lot of time with a perfectly good network barely sending any traffic because the congestion control feedback loop takes too long to ramp the connection back up between outages.</p>
<h2 id="when-tcp-gives-up-on-you">When TCP Gives Up On You</h2>
<p>If we plot the first 0-10% of loss of a connection with 20ms of latency, we wind up with a graph that has this general shape: <sup class="footnote-reference"><a href="#8">5</a></sup>
<img src="../../images/reliable_delivery/tcp_loss_throughput.png" alt="Graph showing TCP Throughput versus packet loss. It is shaped like a backwards hockey stick" /></p>
<p>And the shape of this graph definitely shows the somewhat non-intuitive relationship between Throughput and Packet loss. The first 1% of loss immediately removed 80% of our throughput. Sit with that for a moment. A link that is 99% functional destroys four fifths of your bandwidth. And then it gets worse from there. At 2% loss we lose 90%. From that point on, our throughput is effectively zero. Until it gets to <em>actual</em> zero. And when that happens depends on, as you can probably guess, latency.</p>
<img src="../../images/reliable_delivery/tcp_breaking_point_combined.png" alt="Graph showing when TCP connections at various latencies become nonfunctional" />
<p>You might be wondering why the floor for every request at the bottom of a viable connection is .244 MB/s. This is because TCP is hitting the minimum congestion window and gets throttled by the kernel's minimum RTO. Once you hit this point, extra loss just causes more RTO-driven stalls, but since the throughput can't get any lower, it just flatlines instead. <sup class="footnote-reference"><a href="#9">6</a></sup></p>
<h2 id="when-you-give-up-on-tcp">When You Give Up On TCP</h2>
<p>If you absolutely must send data over a lossy, high latency link, you are going to need to abandon TCP and get creative.</p>
<p>The main limitation of any reliable ordered delivery protocol such as TCP is that it must be polite to its neighbors. TCP must back off aggressively when congestion occurs. If it didn't, a marginal link would immediately become a borderline useless one, because everyone would be a noisy neighbor. Additionally, to them, efficiency does matter a great deal. They won't retransmit data until they know that they have to. Adding additional RTT to this process causes a great deal of pain, as we've previously explained.</p>
<p>But if you absolutely need to guarantee delivery over a marginal link, what can you do? In most cases, you have to trade efficiency for reliability.</p>
<p>Imagine the case where you need to get a control signal to a drone over a connection with 99% frame loss. It's pretty obvious that in order to have any reasonable chance of delivering your message, you first need the data to be smaller than a single frame, and you must send at least 100 of them.</p>
<p>And this is a valid way to handle this issue. If all you need to do is get one message through to the other end (Ideally a small one, like a "Return To Home" command), you can just saturate your entire bandwidth with one-way messages that don't even need to get acked. They will get the message eventually as long as there is signal capture between the radios long enough to transmit a single frame. For cases where the link is degraded, but not entirely jammed, we can figure out the probability that a certain amount of duplication will result in a successful transmission:</p>
<p><strong>Probability of Delivery</strong></p>
<table><thead><tr><th>Copies sent</th><th>10% loss</th><th>50% loss</th><th>75% loss</th><th>90% loss</th><th>99% loss</th></tr></thead><tbody>
<tr><td>1</td><td>90%</td><td>50%</td><td>25%</td><td>10%</td><td>1%</td></tr>
<tr><td>2</td><td>99%</td><td>75%</td><td>44%</td><td>19%</td><td>2%</td></tr>
<tr><td>5</td><td>>99.9%</td><td>97%</td><td>76%</td><td>41%</td><td>5%</td></tr>
<tr><td>10</td><td>~100%</td><td>>99.9%</td><td>94%</td><td>65%</td><td>10%</td></tr>
<tr><td>25</td><td>~100%</td><td>~100%</td><td>>99.9%</td><td>93%</td><td>22%</td></tr>
<tr><td>50</td><td>~100%</td><td>~100%</td><td>~100%</td><td>~99.5%</td><td>40%</td></tr>
<tr><td>100</td><td>~100%</td><td>~100%</td><td>~100%</td><td>~100%</td><td>63%</td></tr>
<tr><td>500</td><td>~100%</td><td>~100%</td><td>~100%</td><td>~100%</td><td>99.3%</td></tr>
</tbody></table>
<p>If you <em>can</em> measure the loss level of your link a simple protocol could just select an appropriate amount of copies from this table. Of course, you still wouldn't have a guaranteed or ordered delivery system. You could also have the receiving side of the connection send ACKs, but now you've given up a significant portion of your bandwidth to guarantee delivery of a message that you could just as effectively ensure by just repeating it ad infinitum until the connection quality increases.</p>
<p>But let's say you need to transmit more than one emergency message? Let's say you have hundreds of sensors generating hundreds of telemetry messages that need to reach the other side? Is there any way you can make this more tolerant to loss while also ensuring some form of reliable ordered delivery?</p>
<h2 id="buffer-stuffing-the-shotgun-approach">Buffer Stuffing: The Shotgun Approach</h2>
<p>In order for this strategy to work, we do have some constraints:</p>
<ul>
<li>Our Maximum Transmission Unit (MTU) is usefully large <sup class="footnote-reference"><a href="#6">7</a></sup></li>
<li>We can fit multiple messages within our MTU (The more, the better)</li>
<li>The underlying link has some kind of Forward Error Correction (FEC) <sup class="footnote-reference"><a href="#5">8</a></sup> that can correct for mid-frame bit-flips <sup class="footnote-reference"><a href="#7">9</a></sup></li>
</ul>
<p>In this toy protocol, we have a <code>Message</code> type that can contain a collection of three different variants of <code>Commands</code>:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>parity_scale_codec::{Decode, Encode};
</span><span>
</span><span style="color:#b48ead;">pub const </span><span style="color:#d08770;">MAX_MESSAGE_SIZE</span><span>: </span><span style="color:#b48ead;">usize </span><span>= </span><span style="color:#d08770;">1500 </span><span>- </span><span style="color:#d08770;">48</span><span>; </span><span style="color:#65737e;">// Roughly MTU minus worst case IPv6 UDP overhead.
</span><span style="color:#b48ead;">pub const </span><span style="color:#d08770;">MAX_PAYLOAD_LEN</span><span>: </span><span style="color:#b48ead;">usize </span><span>= </span><span style="color:#d08770;">128</span><span>;
</span><span>
</span><span style="color:#65737e;">// Per-command encoded size: 1 (variant) + 4 (seq) + 2 (compact len) + 128 (payload) = 135
</span><span style="color:#65737e;">// Message vec prefix: 2 bytes (compact, since capacity > 63)
</span><span style="color:#b48ead;">pub const </span><span style="color:#d08770;">RING_BUF_CAPACITY</span><span>: </span><span style="color:#b48ead;">usize </span><span>= </span><span style="color:#d08770;">MAX_MESSAGE_SIZE</span><span>; </span><span style="color:#65737e;">// Basically how much we can stuff in a single unsegmented datagram
</span><span>
</span><span>#[</span><span style="color:#bf616a;">derive</span><span>(Debug, Clone, PartialEq, Encode, Decode)]
</span><span style="color:#b48ead;">pub enum </span><span>Command {
</span><span> Data { seq: </span><span style="color:#b48ead;">u32</span><span>, payload: Vec<</span><span style="color:#b48ead;">u8</span><span>> },
</span><span> Ack { seq: </span><span style="color:#b48ead;">u32 </span><span>},
</span><span> RetransmitRequest { seq: </span><span style="color:#b48ead;">u32 </span><span>},
</span><span>}
</span><span>
</span><span>#[</span><span style="color:#bf616a;">derive</span><span>(Debug, Clone, PartialEq, Encode, Decode)]
</span><span style="color:#b48ead;">pub struct </span><span>Message {
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">commands</span><span>: Vec<Command>,
</span><span>}
</span></code></pre>
<p>Every time the sender has a new command to send, it packages it into a <code>Message</code> together with the most recent W previously-sent (unacknowledged) commands from a ring buffer. The receiver treats each arriving message as a mini-recovery snapshot: any command it hasn't seen yet gets buffered; any gap at the delivery frontier triggers an explicit retransmit request (NACK).</p>
<p>Because recent commands appear in multiple consecutive outgoing messages, a transient loss rarely requires retransmission at all, since the lost packet's data will simply arrive again inside the next message. In this way, the protocol is very inefficient when it comes to bandwidth, but what it buys us is the ability to skip the TCP-style feedback loop entirely, except in cases of extreme loss.</p>
<h3 id="theoretical-reliability">Theoretical Reliability</h3>
<p>With independent per-message loss probability <strong>p</strong>, a command that appears in <strong>W</strong> consecutive messages goes undelivered only if all W copies are dropped:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>P(undelivered passively) = p^W
</span></code></pre>
<p>For the worst case (128-byte payload, 1500 byte message, W = 10):</p>
<table><thead><tr><th>Loss rate</th><th>p¹⁰</th><th>Failure rate</th></tr></thead><tbody>
<tr><td>1%</td><td>10⁻²⁰</td><td>Vanishingly small</td></tr>
<tr><td>5%</td><td>9.8 × 10⁻¹⁴</td><td>~1 per 10 trillion commands</td></tr>
<tr><td>10%</td><td>10⁻¹⁰</td><td>~1 per 10 billion</td></tr>
<tr><td>20%</td><td>1.0 × 10⁻⁷</td><td>~1 per 10 million</td></tr>
<tr><td>30%</td><td>5.9 × 10⁻⁶</td><td>~1 per 170,000</td></tr>
<tr><td>50%</td><td>9.8 × 10⁻⁴</td><td>~1 per 1,000</td></tr>
</tbody></table>
<p>At 30% loss, passive redundancy alone gives roughly 1 failure per 170,000 commands (in theory). The real-world numbers will look worse, and we'll get to why shortly.</p>
<p>Smaller payloads let you fit more copies per packet, pushing W higher. At 64-byte payloads, W doubles to 20:</p>
<table><thead><tr><th>Loss rate p</th><th>p²⁰</th></tr></thead><tbody>
<tr><td>10%</td><td>10⁻²⁰</td></tr>
<tr><td>20%</td><td>10⁻¹⁴</td></tr>
<tr><td>50%</td><td>10⁻⁶</td></tr>
</tbody></table>
<p>These numbers assume uniform independent loss and, crucially, perfect ACK delivery — neither of which holds in practice. We'll see exactly how that plays out in the benchmarks.</p>
<p>The bandwidth cost scales directly with W, and W scales inversely with payload size. For a 1500-byte MTU, the numbers look like this:</p>
<table><thead><tr><th>Payload</th><th>W</th><th>Overhead</th></tr></thead><tbody>
<tr><td>128 B</td><td>10</td><td>10× bandwidth for the ring window</td></tr>
<tr><td>64 B</td><td>20</td><td>20× bandwidth</td></tr>
<tr><td>32 B</td><td>37</td><td>37× bandwidth</td></tr>
<tr><td>16 B</td><td>63</td><td>63× bandwidth</td></tr>
</tbody></table>
<p>The smaller your payloads, the more copies fit per packet. This is great for reliability but punishing on bandwidth. At 16-byte payloads you get W=63, which is near-perfect reliability, but you're sending 63× the raw data. Whether that's acceptable depends entirely on how much bandwidth you have to spare.</p>
<p>If bandwidth is constrained and loss rates are moderate, the RTO/NACK path can be preferred by reducing W (use larger payloads) or by disabling passive redundancy entirely and relying solely on NACK-driven retransmission. Another option is dynamically adjusting this value from W=1 all the way up to your theoretical maximum based on perceived connection quality. But this adds a lot of complexity. And if you didn't have the bandwidth to sustain high W on a clean link, a lossy one certainly isn't going to give it to you.</p>
<h2 id="benchmarks">Benchmarks</h2>
<p>Piping our toy protocol through <a href="https://crates.io/crates/badnet">badnet</a>, we can simulate a lot of absolutely garbage links.</p>
<p>Imagine a scenario where we have a geostationary satellite with about a 600ms RTT that is experiencing 30% packet loss:</p>
<img src="../../images/reliable_delivery/ring_600ms_30pl.png" alt="Graph showing properties of the Ring toy protocol sending data over a terrible link" />
<p>As you can see, we are sending about 900 <code>Commands</code> per second. Our MTU is 1500, and each command contains 16 bytes of payload (imagine a telemetry counter, or something similar). And I should point out, that TCP would not even function under these circumstances. At 20ms of latency, TCP stopped working at around 17% loss. This connection is <em>significantly</em> worse than that in regards to both packet loss and latency.</p>
<p>With 300ms of delay in each direction, and a packet loss of 30% (10% more than the threshold for TCP becoming non-functional at a lower latency), our passive retransmission strategy allows for extremely reliable delivery:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>--- Summary ---
</span><span>Loss rate: 30.0%
</span><span>RTT (final est.): 500.00ms
</span><span>Commands sent: 10000
</span><span>Commands acked: 10000
</span><span>Commands delivered: 10000
</span><span>Total retransmits: 12
</span><span>OK sent == acked == delivered
</span></code></pre>
<p>We only needed to retransmit 12 commands in total. The astute reader would notice that this is much worse than our theoretical loss probability as mentioned above, at 30% loss we should be having about 1 failure per 170,000 messages (versus the observed ~1 / 1,000). The main reason for this is that we are not duplicating our ACKs. Since we are ack-ing every time we receive a new message, and our acks are cumulative, we usually send enough of them that losing 30% is not problematic mid-stream. The trouble is at the end of the transmission: once we stop receiving new messages, we stop ack-ing. If those final acks are lost, the sender mistakenly assumes the last bits of data were not delivered and the backup retransmission mechanism kicks in.</p>
<p>You can see this more clearly if we crank the loss probability up even higher:</p>
<img src="../../images/reliable_delivery/ring_50p_loss.png" alt="Graph showing properties of the Ring toy protocol sending data over a terrible link" />
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>--- Summary ---
</span><span>Loss rate: 50.0%
</span><span>RTT (final est.): 500.00ms
</span><span>Commands sent: 10000
</span><span>Commands acked: 10000
</span><span>Commands delivered: 10000
</span><span>Total retransmits: 27
</span><span>OK sent == acked == delivered
</span></code></pre>
<p>Despite the loss rate of 50%, our retransmissions mid-stream are nonexistent. However now the tail on the retransmit recovery at the end of the stream is longer, because now only half of our acks make it from the receiver to the transmitter. We could fix this by repeating the last ack at a fixed rate to ensure delivery, but considering this is a toy protocol I think it does a good enough job of demonstrating the concept for now.</p>
<p>And, at the extremes, the protocol can still function. Here is an example of 600ms of latency and <strong>90%</strong> packet loss:</p>
<img src="../../images/reliable_delivery/ring_90p_loss.png" alt="Graph showing properties of the Ring toy protocol sending data over an atrocious link" />
<p>As you can see, we are stalling quite frequently (whenever the window hits 100%, the sender will continually re-transmit the old buffer without adding new items to it until it starts getting acks again). And once again, it's the acks that are the issue here. If we didn't care about reliable delivery, and we were simply tossing the buffer out to the wind and letting what happens happen, the receiver would still succeed in receiving a lot of the commands. Which is definitely something to consider, especially if all you care about is the most recent state of a sensor as opposed to making sure you collect every observation on the remote side. But the most challenging version of this problem is reliable and ordered delivery, which is why I focused on that.</p>
<h2 id="protocol-spec">Protocol Spec</h2>
<p>If the benchmarks have you wondering exactly how the protocol achieves this, here is a more detailed breakdown of what the sender and receiver are actually doing.</p>
<h3 id="sender">Sender</h3>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Sender state
</span><span> ring: Vec<Option<Command>> // circular array of RING_BUF_CAPACITY slots
</span><span> ring_head_seq: u32 // oldest slot still occupied
</span><span> next_seq: u32 // next sequence number to assign
</span><span> unacked: HashMap<u32, (Command, Instant)>
</span></code></pre>
<p>On <code>push_command(payload)</code>:</p>
<ol>
<li>Assign <code>seq = next_seq++</code> and store the command in <code>ring[seq % RING_BUF_CAPACITY]</code>.</li>
<li>If the slot was already occupied (ring wrapped around), evict the old command and advance <code>ring_head_seq</code>.</li>
<li>Compute how many commands fit in one datagram:<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>per_cmd = 1 (variant) + 4 (seq) + 2 (compact length) + payload.len()
</span><span>max_cmds = (MAX_MESSAGE_SIZE − 2) / per_cmd
</span></code></pre>
</li>
<li>Build a snapshot from <code>max(ring_head_seq, seq − max_cmds + 1)</code> through <code>seq</code> inclusive.</li>
<li>Return the <code>Message</code>; the transport layer sends it as a single UDP datagram.</li>
</ol>
<p>On <code>handle_ack(seq)</code>:</p>
<ul>
<li>Remove all commands with sequence number ≤ <code>seq</code> from <code>unacked</code>.</li>
<li>Return the RTT sample for <code>seq</code> (time since last send).</li>
</ul>
<p>On <code>handle_nacks(responses)</code>:</p>
<ul>
<li>For each <code>RetransmitRequest { seq }</code>, look up the command in <code>unacked</code>, reset its send timestamp, and include it in a retransmit <code>Message</code>.</li>
</ul>
<p>On <code>retransmit_aged(threshold)</code>:</p>
<ul>
<li>Find all <code>unacked</code> commands whose last-send timestamp is older than <code>threshold</code> (typically 1.5× smoothed RTT).</li>
<li>Package the oldest <code>max_cmds</code> of them into a single <code>Message</code> and reset their timestamps.</li>
</ul>
<h3 id="receiver">Receiver</h3>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Receiver state
</span><span> next_to_deliver: u32 // highest contiguous seq delivered so far + 1
</span><span> buffer: HashMap<u32, Vec<u8>> // out-of-order data awaiting gap fill
</span><span> highest_seen: Option<u32> // furthest-ahead seq received
</span></code></pre>
<p>On <code>handle_message(msg)</code>:</p>
<ol>
<li>For each <code>Data { seq, payload }</code> in the message:
<ul>
<li>Skip if <code>seq < next_to_deliver</code> (already delivered).</li>
<li>Skip if already buffered.</li>
<li>Otherwise, insert into <code>buffer</code>.</li>
</ul>
</li>
<li>Drain <code>buffer</code> contiguously starting at <code>next_to_deliver</code>, advancing the delivery frontier.</li>
<li>Emit one <code>Ack { seq: next_to_deliver − 1 }</code> (cumulative acknowledgment).</li>
<li>Emit one <code>RetransmitRequest { seq }</code> for each gap between the frontier and <code>highest_seen</code>.</li>
</ol>
<h2 id="prior-art">Prior Art</h2>
<p>This isn't a novel idea. Redundant UDP delivery is a well-worn strategy, particularly in game networking, where low latency matters more than the guarantees TCP provides.</p>
<p>The classic articulation of this is the Tribes Networking Model, described by Mark Frohnmayer and Tim Gift in 2000. Tribes stuffed guaranteed and non-guaranteed messages together into every outgoing packet, using a similar philosophy: cheap redundancy beats expensive retransmission on marginal links.</p>
<p>Modern game engines follow the same playbook. Valve's Steam Datagram Relay and ENet (used in countless indie titles) both implement reliable-over-UDP layers with redundant transmission. Glenn Fiedler's writing on game networking is probably the most cited resource on the topic and is worth reading if this post piqued your interest.</p>
<p>It also shows up outside of games. VoIP and WebRTC use <a href="https://datatracker.ietf.org/doc/html/rfc2198">RFC 2198</a>, a redundancy extension for RTP that piggybacks the previous audio frame onto the current packet — so a single lost packet doesn't produce an audible gap on the other end. Same core idea: accept the bandwidth overhead, eliminate the retransmit round-trip.</p>
<p>The version here is deliberately minimal. Real implementations add packet pacing, more nuanced NACK strategies, and dynamic W adjustment. But the core mechanic is the same: use your spare bandwidth to make loss someone else's problem.</p>
<h2 id="summary">Summary</h2>
<p>TCP is an engineering marvel optimized for the common case: a congested internet where backing off is the polite and correct thing to do. But politeness is a liability when the link itself is the problem. High packet loss triggers congestion control, congestion control shrinks the window, and on a high-latency link the slow recovery from that shrink effectively kills your throughput.</p>
<p>The core insight here is that you have to trade bandwidth for reliability. How you make that trade depends on what you're sending:</p>
<p><strong>One critical message</strong>: just repeat it. Sending 10 copies at 30% loss gives you a failure probability around 1 in 170,000. Sending 25 copies at 90% loss still gets you to 93%. No ACKs, no state, no RTT tax. Just keep repeating it until the link improves, or you have a different message to send.</p>
<p><strong>A stream of small messages</strong>: stuff the recent history into every outgoing packet. A command that rides along in W consecutive messages goes undelivered only if all W are dropped, a p^W event. At 30% loss with W=10, that's roughly 1 failure per 170,000 commands, and in practice the benchmarks show almost no mid-stream retransmissions at all even at 50% loss on a 600ms link.</p>
<p>The catch is overhead. W=10 means you're sending 10× the raw data. If your payloads are small, W climbs quickly. At an MTU of 1500: 16-byte payloads give you W=63, which is excellent for reliability but expensive on a bandwidth-constrained link. The right operating point depends on your payload size, your loss rate, and whether you can afford the retransmit latency of falling back to NACK-driven recovery.</p>
<p>None of this is magic. It's just the math of independent tries working in your favor, plus the observation that some "reliable delivery" problems can be solved by not being stingy with spare bandwidth. Larger byte streams such as firmware updates are a different problem entirely, and one we'll look at next.</p>
<br>
<br>
<br>
<br>
<br>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Explicit Congestion Notifications are probably better, but that is another topic entirely.</p>
</div>
<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
<p>Except when any kind of segmentation occurs.</p>
</div>
<div class="footnote-definition" id="3"><sup class="footnote-definition-label">3</sup>
<p>For the purposes of brevity, I'm going to gloss over the fact that the send buffer and send window are different things and are usually not exactly the same size.</p>
</div>
<div class="footnote-definition" id="4"><sup class="footnote-definition-label">4</sup>
<p>Slow start is a system to...ironically...make scaling up the cwnd faster on new or recovering connections. Once the system is out of slow start and the regular congestion control algorithm takes over, the growth in the cwnd slows significantly on connections with higher RTTs. There are many alternate algorithms and different tunings to reduce the impact of this, but they often have sharp edges and are not the default on most systems and platforms.</p>
</div>
<div class="footnote-definition" id="8"><sup class="footnote-definition-label">5</sup>
<p>If you repeat these experiments on a different computer on the local loopback using TC for traffic disruption as I have done here, you might note that the total throughput will likely change due to differences in memory bandwidth and such, but the important thing to pay attention to about these graphs is the general shape of them, which won't change. Nor will the cutoff for when it ceases to function (latency being equal).</p>
</div>
<div class="footnote-definition" id="9"><sup class="footnote-definition-label">6</sup>
<p>I'm also very sorry for sneaking a logarithmically scaled graph in here.</p>
</div>
<div class="footnote-definition" id="6"><sup class="footnote-definition-label">7</sup>
<p>Definitely not the case with any VHF radios. Many non-commercial UHF radios used for applications like Ardupilot tend to be around 128-512 bytes which is big enough that this <em>might</em> be practical. It's probably more useful for high end IP based systems like Ethernet-bridging radios where the MTU approaches what you might normally see over public internet links (~1500 bytes).</p>
</div>
<div class="footnote-definition" id="5"><sup class="footnote-definition-label">8</sup>
<p>Foreshadowing :)</p>
</div>
<div class="footnote-definition" id="7"><sup class="footnote-definition-label">9</sup>
<p>Otherwise we would need to optimize this system in the other direction: make the frames as small as possible, and send them more frequently.</p>
</div>
Mapping Tomorrow's Crash Sites Today2026-02-04T00:00:00+00:002026-02-04T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/field-finder/<!-- <img src="../../images/field_finder_gis_header.png" alt="A screenshot showing a route and a bunch of green squares" /> -->
<p>In the case of engine failure, how screwed are you?</p>
<p>As of the time of writing, most of my flight experience has been in single-engine aircraft over a particularly flat and reasonably unobstructed part of central Texas. There are a bunch of nice farmers around to maintain emergency runways (though, annoyingly, they do tend to plant crops on them). Doing simulated engine failures and fires over our practice area is pretty hard to mess up. Just point the aircraft in a random direction, and chances are there will be a large flat field there.</p>
<img src="../../images/plenty_of_fields.jpg" alt="Picture taken from a piper archer in an area north of Georgetown showing a ton of fields" />
<p>But, heading back towards our home airport (KGTU), that quickly changes as you cross I-35 and descend to pattern altitude. What was fields only a few miles ago is now densely wooded areas, neighborhoods, and warehouses. At pattern altitude, you have some options (most notably the runway). But taking off from all but one runway, the only realistic options you have below turnback altitude are trying to get to I-35 and its associated medians and clearings (which you probably won't), or landing in the treetops.</p>
<h1 id="a-brief-aside-crash-survivability">A Brief Aside: Crash Survivability</h1>
<p>This is a post about an interesting experiment I did with GIS data, not about aviation safety. But since the subject matter involves forced landings, a small PSA is in order. As someone who's both risk-avoidant and drawn to aviation, I'm naturally inclined to think through worst-case scenarios. I find that understanding the edge cases makes the activity less intimidating, not more. So when the emergency procedure says "select an adequate field for landing," I had to wonder: what happens when there isn't one?</p>
<p>A few years ago, a small plane experienced an engine failure a ways out from KGTU, and landed short of the runway, into a house:</p>
<img src="../../images/plane_in_house.avif" alt="Picture showing a Beech BE35 sticking out of a townhome" />
<p>You might be surprised to learn that the three occupants of this plane survived, sustaining reasonably minor, non-life-threatening injuries. And this illustrates an important point: if you do happen to find yourself in a bad situation where no well manicured perfectly level fields are available, you are far from dead.</p>
<p>Forced landings in inhospitable environments (rough terrain, trees, etc.) are quite survivable <em>when executed properly</em>. Some sources would put the rate of survival somewhere in the 90-97% range <sup class="footnote-reference"><a href="#1">1</a></sup>, which is similar to the survival rate of ditching in water. But ditching in treetops or other inhospitable environments does carry a significant chance of injury. To make things simple, you can assume the chances of a moderate to serious injury when landing in a dense wooded area are probably about 50/50.</p>
<p>Single-engine aircraft stall at low speeds by design, which means lower impact energy when you hit the ground. Certification standards used to mandate low stall speeds for this reason, though these requirements were revised around 2017. This characteristic, combined with structural requirements<sup class="footnote-reference"><a href="#6">2</a></sup> for withstanding abrupt deceleration, means you can land into or onto quite a lot of things with a reasonable chance of survival. Just don't stall and spin. If you lose control and enter a spin, you're likely hitting the ground at 100 knots or more instead of a controlled 50 knots. Get as slow as you can without stalling, and land into the wind if possible. A headwind reduces your groundspeed, which means less energy you will need to dissipate on the ground. And in the words of Bob Hoover, fly the plane as far into the crash as you can. Do this, and your odds of survival are pretty good.</p>
<p>But it is probably going to hurt.</p>
<p>Aircraft engines are surprisingly reliable. So long as you are aware of the risks, and know what needs to be done to survive, I wouldn't judge those who chose to fly over inhospitable terrain in single engine aircraft. After all, you could make an argument that a sizeable portion of Alaska's transportation infrastructure is just that.</p>
<p>But there are also plenty of ways to manage this risk. I like to take a gander in Google Earth and see what was around the airports I fly between to see if there is cause for concern. And for the most part I was just looking for areas with flat land, with no trees or structures. And what follows is really just a stream-lined version of that.</p>
<p>After smashing together a few different publicly available data-sets, I wound up with what I think is a fairly accurate guess of <em>safer</em> forced landing options. And while I was at it, I made a tool to query this data (and it has a few other neat tricks too).</p>
<h1 id="gis-data-sources">GIS Data Sources</h1>
<p>First, we need elevation data. I'm using the <a href="https://www.earthdata.nasa.gov/data/alerts-outages/shuttle-radar-topography-mission-version-3-0-srtm-plus-product-release">SRTMv3</a> dataset. For those who don't know, most of the data in this set came from a really neat space shuttle mission, STS-99, on the Space Shuttle Endeavour. The data was generated via an interferometric synthetic aperture radar attached to the shuttle. </p>
<p>The dataset is available in 3 arcsecond (about 30 meters) or 1 arcsecond (about 10 meters). I'm using the 1 arcsecond DEM simply because I already have it downloaded for another project.</p>
<p>The second dataset we need is the <a href="https://www.usgs.gov/centers/eros/science/national-land-cover-database">National Land Cover Database</a> (NLCD). The NLCD covers the contiguous United States, and provides land cover information at a fairly high resolution.</p>
<p>The NLCD classifies land cover into a number of classes:</p>
<ul>
<li>Open Water</li>
<li>Perennial Ice & Snow</li>
<li>Developed
<ul>
<li>Open Space</li>
<li>Low Intensity</li>
<li>Medium Intensity</li>
<li>High Intensity</li>
</ul>
</li>
<li>Barren Land</li>
<li>Forest
<ul>
<li>Deciduous</li>
<li>Evergreen</li>
<li>Mixed</li>
</ul>
</li>
<li>Shrub/Scrub</li>
<li>Grassland/Herbaceous</li>
<li>Pasture/Hay</li>
<li>Cultivated Crops</li>
<li>Woody Wetlands</li>
<li>Emergent Herbaceous Wetlands</li>
</ul>
<p>This gets us a long way towards our goal, as we can eliminate a lot of options that typically don't make good landing sites (Developed land, Forests, Wetlands). And also, to an extent, prioritize the better options. i.e., Cultivated Crops -> Pasture/Grassland -> Barren. </p>
<p>Lastly, I needed data on airports and navaids. This comes from <a href="https://ourairports.com/data/">ourairports.com</a>. This is, frankly, not the best dataset. It is by far the easiest to use, but there were a lot of inaccuracies for airports near me (I did make an account and correct the ones I knew about, but I'm sure there are more).</p>
<p>I'd love to eventually integrate a more official source, like the FAA ADIP API (assuming I can figure out how to even get access to it).</p>
<h1 id="the-pipeline">The Pipeline</h1>
<p>We have a few goals with our processing pipeline:</p>
<p>We want to find runway-like surfaces. These are surfaces that are flat and free from obstruction. The data we have so far can't guarantee that we will find surfaces free from obstruction, but based on land cover classification, they can give us a decent guess. Part of the goal here is an interactive briefing tool that will allow pilots to inspect potential landing areas, and streamline the process, so this isn't necessarily a problem. Plus, I'm sure I'll think of other ways to further refine this dataset in the future.</p>
<p>So let's focus on the first objective: finding flat areas.</p>
<p>The SRTM dataset is divided into 1°x1° tiles, so these will be our unit of work. I wrote a pipeline-runner binary in rust that will perform each processing step on a per-tile basis. This pipeline writes to various PostGIS tables that represent each step of data refinement, and this allows us to use a lot of native PostGIS functions (which are highly optimized) to help us process our data more easily. We also use Rust <a href="https://gdal.org/en/stable/">GDAL</a> bindings for certain steps as well.</p>
<p>A very easy data source to work with is a raster Digital Elevation Model (DEM). These are usually <code>.tif</code> files (such as in the case of the SRTM3 dataset), though they can really be in any image format (since they are a simple raster). The point is, the band value (usually there is only one band in a DEM, since they usually only encode elevation) corresponds to elevation (and the metadata will help you derive the actual elevation from this value). Here is a render of a DEM .tif from a sampled version of the SRTM3 dataset, centered around Austin, Texas:</p>
<img src="../../images/field_finder/dem_austin.png" alt="Rendering of a DEM file from the SRTM3 dataset showing elevation near Austin, Texas" />
<p>In this rendering, darker areas are lower altitudes, and lighter areas are higher altitudes. One easy thing about working with the SRTM3 data is that the datum is the EGM96 geoid, which approximates Mean Sea level. We would need to be careful if we were working with a dataset using the WGS84 ellipsoid as a datum, as we would need to correct each altitude. Fortunately, aviation charts and such are already using MSL and as such we are in the same conceptual space for altitudes without any conversions (besides feet to meters). For more information on the difference between Ellipsoid and Geoid datums, ESRI has a good explanation <a href="https://www.esri.com/about/newsroom/arcuser/mean-sea-level-gps-geoid">here</a>.</p>
<p>Now we are going to reference this same DEM a few steps down the road, but for now what we care about is finding areas with unacceptable slopes. I picked 8% grade as the rough level of slope I want to exclude (it's configurable, and I'm not sure this is the correct number, but it was a good place to start). I tried this a few different ways, but the way that worked best from a performance standpoint was to take each DEM tile and turn it into an exclusion mask, by comparing adjacent pixels in the raster and figuring out if their slope was greater than 8%. </p>
<p>To get this, we first need to turn the raster of elevations into a raster of slopes:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">/// Generate a slope raster from a DEM using gdaldem
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">generate_slope_raster</span><span><P: AsRef<Path>, Q: AsRef<Path>>(
</span><span> </span><span style="color:#bf616a;">dem_path</span><span>: P,
</span><span> </span><span style="color:#bf616a;">output_path</span><span>: Q,
</span><span>) -> Result<()> {
</span><span> log::info!("</span><span style="color:#a3be8c;">Generating slope raster...</span><span>");
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> output = Command::new("</span><span style="color:#a3be8c;">gdaldem</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">slope</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>(dem_path.</span><span style="color:#96b5b4;">as_ref</span><span>())
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>(output_path.</span><span style="color:#96b5b4;">as_ref</span><span>())
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">-p</span><span>") </span><span style="color:#65737e;">// Output as percentage
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">-compute_edges</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">-of</span><span>").</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">GTiff</span><span>")
</span><span> .</span><span style="color:#96b5b4;">output</span><span>()
</span><span> .</span><span style="color:#96b5b4;">context</span><span>("</span><span style="color:#a3be8c;">Failed to execute gdaldem</span><span>")?;
</span><span>
</span><span> </span><span style="color:#b48ead;">if </span><span>!output.status.</span><span style="color:#96b5b4;">success</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> stderr = String::from_utf8_lossy(&output.stderr);
</span><span> anyhow::bail!("</span><span style="color:#a3be8c;">gdaldem slope failed: {}</span><span>", stderr);
</span><span> }
</span><span>
</span><span> log::info!("</span><span style="color:#a3be8c;">Slope raster generated: {:?}</span><span>", output_path.</span><span style="color:#96b5b4;">as_ref</span><span>());
</span><span> Ok(())
</span><span>}
</span></code></pre>
<p>Then we turn this into a binary mask where 1 are "acceptable areas" with slopes < threshold, and 0 otherwise:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">/// Create a binary slope mask (1 where slope < threshold, 0 otherwise)
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">create_slope_mask</span><span><P: AsRef<Path>, Q: AsRef<Path>>(
</span><span> </span><span style="color:#bf616a;">slope_path</span><span>: P,
</span><span> </span><span style="color:#bf616a;">output_path</span><span>: Q,
</span><span> </span><span style="color:#bf616a;">max_slope_percent</span><span>: </span><span style="color:#b48ead;">f64</span><span>,
</span><span>) -> Result<()> {
</span><span> log::info!("</span><span style="color:#a3be8c;">Creating slope mask (threshold: {}%)...</span><span>", max_slope_percent);
</span><span>
</span><span> </span><span style="color:#65737e;">// Use gdal_calc.py to create binary mask
</span><span> </span><span style="color:#65737e;">// Result is 1 where slope < threshold, 0 otherwise
</span><span> </span><span style="color:#b48ead;">let</span><span> calc_expr = format!("</span><span style="color:#a3be8c;">(A<</span><span style="color:#d08770;">{}</span><span style="color:#a3be8c;">).astype(numpy.uint8)</span><span>", max_slope_percent);
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> output = Command::new("</span><span style="color:#a3be8c;">gdal_calc.py</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">-A</span><span>").</span><span style="color:#96b5b4;">arg</span><span>(slope_path.</span><span style="color:#96b5b4;">as_ref</span><span>())
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>(format!("</span><span style="color:#a3be8c;">--outfile=</span><span style="color:#d08770;">{}</span><span>", output_path.</span><span style="color:#96b5b4;">as_ref</span><span>().</span><span style="color:#96b5b4;">display</span><span>()))
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>(format!("</span><span style="color:#a3be8c;">--calc=</span><span style="color:#d08770;">{}</span><span>", calc_expr))
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--type=Byte</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--NoDataValue=0</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--quiet</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--overwrite</span><span>")
</span><span> .</span><span style="color:#96b5b4;">output</span><span>()
</span><span> .</span><span style="color:#96b5b4;">context</span><span>("</span><span style="color:#a3be8c;">Failed to execute gdal_calc.py</span><span>")?;
</span><span>
</span><span> </span><span style="color:#b48ead;">if </span><span>!output.status.</span><span style="color:#96b5b4;">success</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> stderr = String::from_utf8_lossy(&output.stderr);
</span><span> anyhow::bail!("</span><span style="color:#a3be8c;">gdal_calc.py mask creation failed: {}</span><span>", stderr);
</span><span> }
</span><span>
</span><span> log::info!("</span><span style="color:#a3be8c;">Slope mask created: {:?}</span><span>", output_path.</span><span style="color:#96b5b4;">as_ref</span><span>());
</span><span> Ok(())
</span><span>}
</span></code></pre>
<p>What you wind up with, is something that looks like this:</p>
<img src="../../images/field_finder/slope_mask.png" alt="Rendering of an example slope mask near Austin, Texas" />
<p>This gives us a very efficient way to exclude areas from the next step while we are still working with rasters.</p>
<p>So we've taken care of flat, now we need to move on to the next thing: ground cover.</p>
<p>To start, we're going to take the NLCD (which is also a raster) and multiply it by our mask. Since any value with an unacceptable slope will be a "0" value in the mask, those areas will be effectively eliminated from the NLCD raster:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">/// Apply mask to raster (multiply raster by mask)
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">apply_mask_to_raster</span><span><P: AsRef<Path>, Q: AsRef<Path>, R: AsRef<Path>>(
</span><span> </span><span style="color:#bf616a;">raster_path</span><span>: P,
</span><span> </span><span style="color:#bf616a;">mask_path</span><span>: Q,
</span><span> </span><span style="color:#bf616a;">output_path</span><span>: R,
</span><span>) -> Result<()> {
</span><span> log::info!("</span><span style="color:#a3be8c;">Applying mask to NLCD...</span><span>");
</span><span>
</span><span> </span><span style="color:#65737e;">// Use gdal_calc.py to multiply raster by mask
</span><span> </span><span style="color:#65737e;">// Where mask is 0 (steep slope), result will be 0 (NoData for NLCD)
</span><span> </span><span style="color:#b48ead;">let</span><span> output = Command::new("</span><span style="color:#a3be8c;">gdal_calc.py</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">-A</span><span>").</span><span style="color:#96b5b4;">arg</span><span>(raster_path.</span><span style="color:#96b5b4;">as_ref</span><span>())
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">-B</span><span>").</span><span style="color:#96b5b4;">arg</span><span>(mask_path.</span><span style="color:#96b5b4;">as_ref</span><span>())
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>(format!("</span><span style="color:#a3be8c;">--outfile=</span><span style="color:#d08770;">{}</span><span>", output_path.</span><span style="color:#96b5b4;">as_ref</span><span>().</span><span style="color:#96b5b4;">display</span><span>()))
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--calc=A*B</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--type=Byte</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--NoDataValue=0</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--quiet</span><span>")
</span><span> .</span><span style="color:#96b5b4;">arg</span><span>("</span><span style="color:#a3be8c;">--overwrite</span><span>")
</span><span> .</span><span style="color:#96b5b4;">output</span><span>()
</span><span> .</span><span style="color:#96b5b4;">context</span><span>("</span><span style="color:#a3be8c;">Failed to execute gdal_calc.py</span><span>")?;
</span><span>
</span><span> </span><span style="color:#b48ead;">if </span><span>!output.status.</span><span style="color:#96b5b4;">success</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> stderr = String::from_utf8_lossy(&output.stderr);
</span><span> anyhow::bail!("</span><span style="color:#a3be8c;">gdal_calc.py mask application failed: {}</span><span>", stderr);
</span><span> }
</span><span>
</span><span> log::info!("</span><span style="color:#a3be8c;">Mask applied: {:?}</span><span>", output_path.</span><span style="color:#96b5b4;">as_ref</span><span>());
</span><span> Ok(())
</span><span>}
</span></code></pre>
<p>Now what we have is an NLCD raster with empty areas where the slope was too great. But to serve this efficiently to clients, we should deliver this data as individual polygons in GeoJSON as opposed to an image. This would allow us to preserve the accuracy of the dataset while also optimizing how we query and further refine it.</p>
<p>I'm going to gloss over this next part—apologies in advance. To put it simply, I'm using the <code>GDALPolygonize</code> FFI from the <a href="https://docs.rs/gdal-sys/0.12.0/gdal_sys/">gdal_sys</a> crate, but there is a lot more to it than that unfortunately and I'm not going to unpack it here. The good news is, you can get a decent result just by running the function on a raster. But if you want to optimize the resultant geometries for easy querying/spatial indexing/processing, you are going to spend a lot of time figuring out sieve filters and subdividing large contiguous blocks of land when appropriate. None of it is hard, but it is a bit tedious and requires a lot of experimentation. </p>
<p>But what I eventually wound up with, was an NLCD vector that looks somewhat like this:</p>
<img src="../../images/field_finder/nlcd_vector.png" alt="Rendering of a vectorized NLCD dataset near Austin, Texas" />
<p>Now we need to figure out the dimensions of our newly generated vectors. Specifically, we are looking for at least a 2500ft long 50ft wide path contained within each vector for it to qualify as a potentially landable area.</p>
<p>To do this, we are going to sample 200 points from the polygon's geometry, and check each pair to see if:</p>
<ul>
<li>It is longer than the minimum candidate distance (currently 2500 feet, but configurable)</li>
<li>It is longer than the best one we've already found</li>
<li>Verify that the entire length of the segment is entirely contained within the geometry of the polygon</li>
</ul>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">/// Find the longest line segment entirely contained within a single polygon
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">find_longest_interior_segment</span><span>(
</span><span> </span><span style="color:#bf616a;">polygon</span><span>: &Polygon<</span><span style="color:#b48ead;">f64</span><span>>,
</span><span> </span><span style="color:#bf616a;">min_width_degrees</span><span>: </span><span style="color:#b48ead;">f64</span><span>,
</span><span>) -> Result<Option<(LineString<</span><span style="color:#b48ead;">f64</span><span>>, </span><span style="color:#b48ead;">f64</span><span>)>> {
</span><span> </span><span style="color:#65737e;">// Sample points along the polygon boundary
</span><span> </span><span style="color:#b48ead;">let</span><span> boundary_points = </span><span style="color:#96b5b4;">sample_boundary_points</span><span>(polygon, </span><span style="color:#d08770;">BOUNDARY_SAMPLES</span><span>)?;
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> boundary_points.</span><span style="color:#96b5b4;">len</span><span>() < </span><span style="color:#d08770;">2 </span><span>{
</span><span> </span><span style="color:#b48ead;">return </span><span>Ok(None);
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">let mut</span><span> best_segment: Option<LineString<</span><span style="color:#b48ead;">f64</span><span>>> = None;
</span><span> </span><span style="color:#b48ead;">let mut</span><span> best_length = </span><span style="color:#d08770;">0.0</span><span>;
</span><span>
</span><span> </span><span style="color:#65737e;">// Check all pairs of boundary points
</span><span> </span><span style="color:#b48ead;">for </span><span>(i, p1) in boundary_points.</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">enumerate</span><span>() {
</span><span> </span><span style="color:#b48ead;">for</span><span> p2 in boundary_points.</span><span style="color:#96b5b4;">iter</span><span>().</span><span style="color:#96b5b4;">skip</span><span>(i + </span><span style="color:#d08770;">1</span><span>) {
</span><span> </span><span style="color:#65737e;">// Skip points that are too close together
</span><span> </span><span style="color:#b48ead;">let</span><span> pair_distance = </span><span style="color:#96b5b4;">distance</span><span>(p1, p2);
</span><span> </span><span style="color:#b48ead;">if</span><span> pair_distance < </span><span style="color:#d08770;">MIN_CANDIDATE_DISTANCE </span><span>{
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// Skip if this pair can't possibly beat our best
</span><span> </span><span style="color:#b48ead;">if</span><span> pair_distance <= best_length {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// Create line segment between these boundary points
</span><span> </span><span style="color:#b48ead;">let</span><span> segment = LineString::from(vec![*p1, *p2]);
</span><span>
</span><span> </span><span style="color:#65737e;">// Check if the segment is entirely within the polygon
</span><span> </span><span style="color:#65737e;">// We use a sampling approach since geo's Contains can be strict
</span><span> </span><span style="color:#b48ead;">if </span><span style="color:#96b5b4;">is_segment_inside_polygon</span><span>(&segment, polygon) {
</span><span> </span><span style="color:#b48ead;">let</span><span> length = segment.</span><span style="color:#96b5b4;">euclidean_length</span><span>();
</span><span> </span><span style="color:#b48ead;">if</span><span> length > best_length {
</span><span> </span><span style="color:#65737e;">// Check width requirement by verifying perpendicular clearance
</span><span> </span><span style="color:#b48ead;">if </span><span style="color:#96b5b4;">check_width_requirement</span><span>(&segment, polygon, min_width_degrees) {
</span><span> best_length = length;
</span><span> best_segment = Some(segment);
</span><span> }
</span><span> }
</span><span> }
</span><span> }
</span><span> }
</span><span>
</span><span> Ok(best_segment.</span><span style="color:#96b5b4;">map</span><span>(|</span><span style="color:#bf616a;">s</span><span>| (s, best_length)))
</span><span>}
</span></code></pre>
<p>And we can query the intermediate table with QGIS to see the output of this step:</p>
<img src="../../images/field_finder/landable_lengths.png" alt="Rendering of potentially landable areas with the landable lengths rendered within them" />
<p>This is, unfortunately, O(N²) at best. I had considered doing a rotating caliper or indexing by antipodals, but this would only work for mostly square shaped polygons. The current sampling plus brute force method handles shapes with holes in their internal geometry fairly well. And I'm not sure either of the other options would be as good:</p>
<img src="../../images/field_finder/internal_geometry.png" alt="A landable area with some un-landable-areas inside it" />
<p>Moreover, I don't mind spending the extra CPU cycles on this pipeline, because it will only be re-run whenever I change something about it, or when the datasets update (at most, annually). So it's really good enough as is, and the tradeoff is well worth it.</p>
<p>When zoomed out, these segments create an interesting visualization that represents how contiguous the land cover areas are. Areas with more segments have more varied land cover, while long segments indicate areas of contiguous land cover.</p>
<p>For instance, here is a portion of the Texas gulf coast<sup class="footnote-reference"><a href="#2">3</a></sup>:</p>
<img src="../../images/field_finder/texas_gulf_coast_landable_segments.png" alt="Rendering of potentially landable along the Texas gulf coast" />
<p>For those curious, the large area devoid of any landable segments in the top left of the map are the heavily urban parts of San Antonio.</p>
<p>As previously mentioned, the unit of work is a 1x1 degree DEM raster, so we can iterate through this pipeline in small chunks. I've picked Texas as a test area (mostly because I'm familiar with the local geography). The dashboard below shows the progress of each tile's pipeline:</p>
<img src="../../images/field_finder/segment_processing_dashboard.png" alt="Dashboard showing tile processing progress" />
<p>And I'll go more into how the frontend works another day, but here is what the current (very primitive) interface looks like:</p>
<img src="../../images/field_finder/kgtu_departure.png" alt="Image showing UI from the departure airport with landable areas" />
<p>We have a place to enter a route string (it isn't correctly parsing all routes yet, but if your route only includes airfields and navaids in the Ourairports dataset, it should work fine). The line is rendered from the departure airport to the destination airport and is colored as so:</p>
<ul>
<li>Magenta, if you can glide to a runway</li>
<li>Green, if you can glide to a landable_area <sup class="footnote-reference"><a href="#3">4</a></sup></li>
<li>Red, if you can't glide to any landable_area</li>
</ul>
<p>Additionally, we are rendering an overlay of the actual landable areas. Green ones are ones we can reach at some point on our journey, Grey ones are out of range, but shown for context (i.e. you might choose to cruise at a higher altitude to reach a landing area farther away). </p>
<p>But most importantly, this tool makes it easy to find areas where landable areas are sparse, and then visually check satellite imagery to see if they are viable or not<sup class="footnote-reference"><a href="#4">5</a></sup>:</p>
<img src="../../images/field_finder/kgtu_zoomed.png" alt="Image showing UI zoomed in, so you can see the landable area outlines over satellite imagery." />
<p>And, in this case at least, it does a pretty good job of highlighting areas that are viable in an otherwise urban/forested area. You will notice that there are some fields that aren't highlighted, but remember, we are rejecting any area without a 2500x50ft flat area.</p>
<p>Looking at the arrival area, we can see an area of the flight where we would not be able to glide to any options, as we cross the Gulf Intracoastal Waterway:</p>
<img src="../../images/field_finder/kras_arrival.png" alt="Image showing a stretch of the flight plan where we don't have any good ditching options over water." />
<p>And the tool allows us to play some "what if" scenarios. In the original plan we had a pretty lazy 500fpm descent, what if we increase that to 800fpm? Sure enough, this gets rid of this period of risk and we can now glide to several options even as we are in the middle of the waterway.</p>
<img src="../../images/field_finder/kras_800fpm.png" alt="Image showing a green segment where a red one previously was due to an increased rate of descent." />
<p>You will notice there is a small red segment right as we get to the airport, this is there for several reasons. One, we don't assume that we arrive at the airport at pattern altitude to fly a pattern. I plan on eventually adding pattern entry logic to the <a href="https://pattern-visualizer.graysonhead.net">Pattern Visualizer</a>, and that logic will eventually make its way into this app as well.</p>
<p>Second, we are cheating a bit with how we calculate what we can glide to. For the purposes of making the app more performant, we are including a landing area only if we can glide to the centroid of that area. For landing areas, this is conservative, and fine. But with Runways which are long and skinny, we really only care if we can hit the threshold (since we know for sure that runways are usually pretty good places to land). </p>
<p>It's also useful for answering a question I like to ask before I do a night cross-country: What is the minimum altitude at which you can fly a given route and always (or at least mostly) be in range of a runway?</p>
<p>In the case of a flight between KGTU and KHYI, 6,500 feet leaves you at risk for only two short segments<sup class="footnote-reference"><a href="#5">6</a></sup>.</p>
<img src="../../images/field_finder/kgtu_khyi.png" alt="UI showing a flight between KGTU and KHYI." />
<h1 id="problems-and-future-improvements">Problems and Future Improvements</h1>
<p>This is obviously imperfect, and by no means done. Some areas we consider landable right now are terrible options. For instance, just looking around Austin:</p>
<p>We have a forest, which is misclassified as a field:</p>
<img src="../../images/field_finder/forested_field.png" alt="What is clearly a forest is misclassified as a field" />
<p>A lot of perfectly good fields are bisected by a line of trees:</p>
<img src="../../images/field_finder/tree_bisected.png" alt="Two fields seperated by a line of trees" />
<p>In this case, either of the fields would probably make an acceptable forced landing site. But it would be nice if we could post-analyze each tile and see if the landable segment crosses any interesting objects. I'm curious if I could use something like tensorflow to process the satellite imagery for each segment and see if the imagery shows obstructions that don't show up in the NLCD dataset.</p>
<p>And here, we have a very long perfectly runway shaped field, but unfortunately this field is cleared because its occupied by high voltage power lines.</p>
<img src="../../images/field_finder/power_lines.png" alt="A field occupied by high voltage power lines." />
<p>This is a partially solvable problem, as there are several datasets that can be combined to create linesegments for most high voltage power lines in the contiguous US. We could do something similar to (or combined with) our slope mask earlier. Take these vectors, convert them to a true/false raster (adding some buffer depending on the height of the towers), and then use that as an additional mask (or merge it with the existing one). It won't get all power lines, but it should get most of the big ones.</p>
<p>Additionally, the one huge thing that is missing here is winds. A lot of the existing code was designed with winds in mind, but I'm honestly not sure how I want to implement that from a UI perspective yet.</p>
<h1 id="closing">Closing</h1>
<p>This tool won't tell you whether a field is safe to land in. Only your eyes and judgment can do that. For me, the results aren't a go/no-go decision. They're just an easy way to load this aspect of the flight into my head during preflight. The tool highlights where options thin out, what altitudes might be worth considering, and which parts of the route deserve extra attention. And it can answer questions that would be tedious to answer manually, like what altitude you need to maintain to stay within gliding range of a runway.</p>
<p>The datasets here are massive (the SRTM and NLCD data cover the entire continental US at high resolution), which made the pipeline work particularly satisfying. There's something deeply appealing about turning gigabytes of elevation and land cover data into what I hope will be actionable flight planning information.</p>
<p>This is very much a work in progress. The power line masking needs implementation, the classification accuracy could improve with some ML-based satellite imagery analysis, etc. And I haven't even looked into the practicality of using LIDAR data in urban areas (it turns out, there are a lot of public lidar datasets). I'll likely revisit this project as I refine the algorithms and expand coverage.</p>
<p>I'd like to eventually host this somewhere publicly accessible, but I haven't worked out how to do that without it becoming a money pit. The infrastructure itself is straightforward—PostGIS database, API server, caching layer—but serving vector tiles for the entire continental US to an unknown number of users means either paying for capacity I don't need most days, or getting surprised by a bill when someone posts it on Reddit. For now, I'm focused on getting the data pipeline and algorithms right. Deployment can wait until I figure out sustainable hosting.</p>
<br>
<br>
<br>
<br>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>https://aviationsafetymagazine.com/features/water-or-trees/</p>
</div>
<div class="footnote-definition" id="6"><sup class="footnote-definition-label">2</sup>
<p>https://www.ecfr.gov/current/title-14/chapter-I/subchapter-C/part-23/subpart-C/subject-group-ECFR72b1233739a7fce/section-23.2270</p>
</div>
<div class="footnote-definition" id="2"><sup class="footnote-definition-label">3</sup>
<p>You will also notice that there is a max size for contiguous chunks of land of about 2x2km. These smaller chunks of land are just easier to work with, and the subdivision is more or less a thin wrapper around the PostGIS ST_SUBDIVIDE function, so I'm glossing over it.</p>
</div>
<div class="footnote-definition" id="3"><sup class="footnote-definition-label">4</sup>
<p>Calling these landable_areas to the user is not a good idea, but "suspected landable area" doesn't quite roll off the tongue. I'm not sure what to call them, but that is a problem for future Grayson</p>
</div>
<div class="footnote-definition" id="4"><sup class="footnote-definition-label">5</sup>
<p>Obviously the satellite imagery could be out of date, but the only way you are gonna fix that is to go fly over it yourself, at which point you've already accepted the risk anyways.</p>
</div>
<div class="footnote-definition" id="5"><sup class="footnote-definition-label">6</sup>
<p>One of the fields providing this coverage is private and probably not lit, but that could be filtered out with a better airport datasource. And also, as yet another reminder, this doesn't take winds into account at all yet.</p>
</div>
Bevy Gravity2025-11-30T00:00:00+00:002025-11-30T00:00:00+00:00Unknownhttps://blog.graysonhead.net/projects/bevy-gravity/<h2 id="project-description">Project Description</h2>
<p>This is an interactive WASM demo of <a href="https://github.com/graysonhead/bevy-gravity">bevy-gravity</a></p>
<p>See the <a href="../../posts/bevy-gravity/">post</a> for more info.</p>
<h2 id="demo">Demo</h2>
<div class="info">
<p style="margin-top: 15px; font-size: 14px; color: #aaa;">
Press F1 in-game to view controls
</p>
</div>
<div id="loading" class="loading">
<h2>Loading gravity simulation...</h2>
<p>Please wait while the WASM module loads.</p>
</div>
<p><canvas id="bevy"></canvas></p>
<script type="module">
import init, { run_two_bodies } from '../../wasm/gravity/gravity_wasm.js';
async function run() {
try {
await init();
run_two_bodies();
document.getElementById('loading').style.display = 'none';
} catch (e) {
console.error('Failed to load WASM:', e);
document.getElementById('loading').innerHTML = '<h2>Error loading simulation</h2><p>Please check the console for details.</p>';
}
}
run();
</script>Bevy Systems: Gravity2025-11-14T00:00:00+00:002025-11-14T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/bevy-gravity/<p><canvas id="bevy-demo"></canvas></p>
<script type="module">
import init, { run_demo } from '../../wasm/gravity/demo/gravity_wasm.js';
async function run() {
try {
await init();
run_demo();
document.getElementById('loading').style.display = 'none';
} catch (e) {
console.error('Failed to load WASM:', e);
document.getElementById('loading').innerHTML = '<h2>Error loading simulation</h2><p>Please check the console for details.</p>';
}
}
run();
</script>
<p>I haven't had a lot of free time lately, but my desire to start a million projects I'll never finish has in no way diminished. I've been thinking about these projects during car rides, birthday parties, and those 3am wake-ups where I suddenly realize a better way to do something I haven't touched in 5 years.</p>
<p>After watching <a href="https://www.youtube.com/watch?v=QPuIysZxXwM">Make Systems Not Games</a>, I decided to focus on building modular, reusable systems instead of complete game prototypes. This is one of those systems. It was quick to build. But only because I'd failed at it enough times before to know what problems I needed to solve. (And because Claude handled the easy parts, plus some hard parts with coaching.)</p>
<h2 id="what-this-is">What This Is</h2>
<p>This is an <strong>n-body gravity simulator</strong> for Bevy. Not the gravity you'd find in a physics engine like <a href="https://github.com/dimforge/rapier">Rapier</a>, but the kind you'd find in Kerbal Space Program or Universe Sandbox. The kind where you fly a spaceship around planets and have to reason about orbital mechanics and delta-v budgets like some kind of masochist who enjoys pain (or has a degree in aerospace engineering, or just played a lot of KSP).</p>
<p>This post walks through three key challenges I had to solve:</p>
<ol>
<li><strong>Scale and Units</strong>: How do you simulate both planetary orbits and surface-level physics without floating-point errors destroying everything?</li>
<li><strong>Integration Accuracy</strong>: How do you keep orbits stable over thousands of simulation steps without them decaying into chaos?</li>
<li><strong>Time Acceleration</strong>: How do you speed up time 1000x without breaking the physics?</li>
</ol>
<p>There are limitations to doing this in realtime. Some of which I could fix with CUDA, but that would turn a 4-day project into a 4-week project. What's here is "Good Enough" for the scale I'm considering.</p>
<p>Let's dive in.</p>
<h2 id="challenge-1-scale-and-units">Challenge 1: Scale and Units</h2>
<p>The first problem with gravity simulations is scale. You might want to simulate:</p>
<ul>
<li>Surface-level physics (meters and kilograms)</li>
<li>Planetary systems (thousands of kilometers, Earth masses)</li>
<li>Solar systems (astronomical units, solar masses)</li>
</ul>
<p>If you use real-world values directly, you'll end up with floating-point precision errors and numbers like <code>1.989e30</code> scattered throughout your code. The solution is a configurable unit system that lets you define what "1 unit" means in your simulation:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">derive</span><span>(Debug, Clone, Resource)]
</span><span style="color:#b48ead;">pub struct </span><span>GravityConfig {
</span><span> </span><span style="color:#65737e;">/// Length scale defining how real-world distances map to game coordinates.
</span><span> </span><span style="color:#65737e;">///
</span><span> </span><span style="color:#65737e;">/// Examples:
</span><span> </span><span style="color:#65737e;">/// - `Length::from_meters(1.0)`: 1 Bevy unit = 1 meter (surface scale)
</span><span> </span><span style="color:#65737e;">/// - `Length::from_kilometers(1.0)`: 1 Bevy unit = 1 km (regional scale)
</span><span> </span><span style="color:#65737e;">/// - `Length::from_au(0.01)`: 1 Bevy unit = 0.01 AU (solar system scale)
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">length_scale</span><span>: Length,
</span><span>
</span><span> </span><span style="color:#65737e;">/// Mass scale allowing scaling masses to avoid very large/small numbers.
</span><span> </span><span style="color:#65737e;">///
</span><span> </span><span style="color:#65737e;">/// Examples:
</span><span> </span><span style="color:#65737e;">/// - `Mass::from_kg(1.0)`: Mass values represent kilograms directly
</span><span> </span><span style="color:#65737e;">/// - `Mass::earth()`: Mass values represent Earth masses
</span><span> </span><span style="color:#65737e;">/// - `Mass::from_kg(1.989e30)`: Mass values represent Solar masses
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">mass_scale</span><span>: Mass,
</span><span>
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>GravityConfig {
</span><span> </span><span style="color:#65737e;">/// Configuration optimized for solar system simulations.
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">solar_system</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> length_scale: Length::from_au(</span><span style="color:#d08770;">0.01</span><span>), </span><span style="color:#65737e;">// 0.01 AU per unit
</span><span> mass_scale: Mass::earth(), </span><span style="color:#65737e;">// Earth masses
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Configuration optimized for planetary system simulations (Earth-Moon scale).
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">planetary</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> length_scale: Length::from_kilometers(</span><span style="color:#d08770;">1000.0</span><span>), </span><span style="color:#65737e;">// 1000 km per unit
</span><span> mass_scale: Mass::earth(), </span><span style="color:#65737e;">// Earth masses
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Configuration optimized for atmospheric and near-surface operations.
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">atmospheric</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> length_scale: Length::from_kilometers(</span><span style="color:#d08770;">1.0</span><span>), </span><span style="color:#65737e;">// 1 km per unit
</span><span> mass_scale: Mass::earth(), </span><span style="color:#65737e;">// Earth masses
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Configuration optimized for surface operations and small-scale physics.
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">surface</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> length_scale: Length::from_meters(</span><span style="color:#d08770;">1.0</span><span>), </span><span style="color:#65737e;">// 1 meter per unit
</span><span> mass_scale: Mass::from_kg(</span><span style="color:#d08770;">1.0</span><span>), </span><span style="color:#65737e;">// 1 kg per unit
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Get the effective gravitational constant for this scale configuration.
</span><span> </span><span style="color:#65737e;">///
</span><span> </span><span style="color:#65737e;">/// The gravitational constant is scaled according to the unit conversions:
</span><span> </span><span style="color:#65737e;">/// G_effective = G_real * (mass_scale) / (length_scale³)
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">gravitational_constant</span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> </span><span style="color:#b48ead;">f32 </span><span>{
</span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">G_SI</span><span>: </span><span style="color:#b48ead;">f32 </span><span>= </span><span style="color:#d08770;">6.674e-11</span><span>; </span><span style="color:#65737e;">// m³/(kg⋅s²)
</span><span>
</span><span> </span><span style="color:#65737e;">// Scale conversions for space and mass units
</span><span> </span><span style="color:#d08770;">G_SI </span><span>* </span><span style="color:#bf616a;">self</span><span>.mass_scale.</span><span style="color:#96b5b4;">kg</span><span>() / </span><span style="color:#bf616a;">self</span><span>.length_scale.</span><span style="color:#96b5b4;">meters</span><span>().</span><span style="color:#96b5b4;">powi</span><span>(</span><span style="color:#d08770;">3</span><span>)
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Convert a real-world distance (in meters) to Bevy units.
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">meters_to_units</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">meters</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> </span><span style="color:#b48ead;">f32 </span><span>{
</span><span> meters / </span><span style="color:#bf616a;">self</span><span>.length_scale.</span><span style="color:#96b5b4;">meters</span><span>()
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Convert Bevy units to real-world distance (in meters).
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">units_to_meters</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">units</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> </span><span style="color:#b48ead;">f32 </span><span>{
</span><span> units * </span><span style="color:#bf616a;">self</span><span>.length_scale.</span><span style="color:#96b5b4;">meters</span><span>()
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Convert a real-world mass (in kg) to simulation mass units.
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">kg_to_mass_units</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">kg</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> </span><span style="color:#b48ead;">f32 </span><span>{
</span><span> kg / </span><span style="color:#bf616a;">self</span><span>.mass_scale.</span><span style="color:#96b5b4;">kg</span><span>()
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Convert simulation mass units to real-world mass (in kg).
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">mass_units_to_kg</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">mass_units</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> </span><span style="color:#b48ead;">f32 </span><span>{
</span><span> mass_units * </span><span style="color:#bf616a;">self</span><span>.mass_scale.</span><span style="color:#96b5b4;">kg</span><span>()
</span><span> }
</span><span>}
</span></code></pre>
<p>This was the missing piece in my previous failed attempts. Different games need different scales; a full-scale solar system simulation with 1:1 timescale would be incredibly boring. This config lets you tune the simulation while preserving physical accuracy.</p>
<p>I also wanted type-safe units throughout the simulation. The challenge is that Bevy's reflection system needs to understand these values. UOM's units don't implement <code>Reflect</code>, so I created wrapper components that convert between type-safe units and Bevy's internal representation.</p>
<h3 id="components">Components</h3>
<p>With the scale system in place, entities need three things:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">derive</span><span>(Reflect, Component, Default)]
</span><span>#[</span><span style="color:#bf616a;">reflect</span><span>(Component)]
</span><span style="color:#b48ead;">pub struct </span><span>CelestialBody {
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">velocity</span><span>: Vec3,
</span><span>}
</span><span>
</span><span style="color:#65737e;">/// Marker component for bodies that should remain fixed in position.
</span><span style="color:#65737e;">/// Pinned bodies are immune to gravitational forces but still exert gravitational influence on other bodies.
</span><span>#[</span><span style="color:#bf616a;">derive</span><span>(Reflect, Component, Default)]
</span><span>#[</span><span style="color:#bf616a;">reflect</span><span>(Component)]
</span><span style="color:#b48ead;">pub struct </span><span>Pinned;
</span><span>
</span><span>
</span><span style="color:#65737e;">/// Mass component with type-safe units that supports Bevy reflection.
</span><span style="color:#65737e;">///
</span><span style="color:#65737e;">/// This wraps the UOM Mass type to provide compile-time unit safety while
</span><span style="color:#65737e;">/// maintaining compatibility with Bevy's reflection system.
</span><span>#[</span><span style="color:#bf616a;">derive</span><span>(Component, Reflect, Clone, Debug)]
</span><span>#[</span><span style="color:#bf616a;">reflect</span><span>(Component)]
</span><span style="color:#b48ead;">pub struct </span><span>Mass {
</span><span> </span><span style="color:#65737e;">/// Raw mass value in kilograms (for reflection/serialization)
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">kilograms</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#65737e;">/// Type-safe mass quantity (ignored by reflection to avoid UOM issues)
</span><span> #[</span><span style="color:#bf616a;">reflect</span><span>(ignore)]
</span><span> #[</span><span style="color:#bf616a;">allow</span><span>(dead_code)]
</span><span> </span><span style="color:#bf616a;">inner</span><span>: uom::si::f32::Mass,
</span><span>}
</span></code></pre>
<p>Entities need a <code>CelestialBody</code> component to participate in gravity, a <code>Mass</code> to know how much gravitational influence they have, and optionally <code>Pinned</code> if they should stay fixed (like a sun that doesn't move but still pulls on planets).</p>
<h2 id="challenge-2-integration-accuracy">Challenge 2: Integration Accuracy</h2>
<p>Now for the tricky part: actually simulating orbits over time. The basic gravity calculation is straightforward:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">calculate_velocity_impact</span><span>(
</span><span> </span><span style="color:#bf616a;">our_transform</span><span>: &GlobalTransform,
</span><span> </span><span style="color:#bf616a;">our_mass</span><span>: &Mass,
</span><span> </span><span style="color:#bf616a;">other_transform</span><span>: &GlobalTransform,
</span><span> </span><span style="color:#bf616a;">other_mass</span><span>: &Mass,
</span><span> </span><span style="color:#bf616a;">time_step_secs</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span>) -> Vec3 {
</span><span> </span><span style="color:#b48ead;">let</span><span> our_body_transform = our_transform.</span><span style="color:#96b5b4;">translation</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> other_body_transform = other_transform.</span><span style="color:#96b5b4;">translation</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> distance = our_body_transform.</span><span style="color:#96b5b4;">distance</span><span>(other_body_transform);
</span><span> </span><span style="color:#b48ead;">let</span><span> direction = (other_body_transform - our_body_transform).</span><span style="color:#96b5b4;">normalize</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> force_magnitude = </span><span style="color:#d08770;">GRAVITY </span><span>* (our_mass.mass * other_mass.mass) / distance.</span><span style="color:#96b5b4;">powi</span><span>(</span><span style="color:#d08770;">2</span><span>);
</span><span> </span><span style="color:#b48ead;">let</span><span> force = direction * force_magnitude;
</span><span> </span><span style="color:#b48ead;">let</span><span> acceleration = force / our_mass.mass;
</span><span> acceleration * time_step_secs
</span><span>}
</span></code></pre>
<p>Then you need a system to move the object based on its velocity:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">update_position</span><span>(
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">bodies</span><span>: Query<(&CelestialBody, &</span><span style="color:#b48ead;">mut</span><span> Transform)>,
</span><span> </span><span style="color:#bf616a;">time</span><span>: Res<Time>,
</span><span>) {
</span><span> </span><span style="color:#b48ead;">for </span><span>(body, </span><span style="color:#b48ead;">mut</span><span> transform) in &</span><span style="color:#b48ead;">mut</span><span> bodies {
</span><span> transform.translation -= body.velocity * time.</span><span style="color:#96b5b4;">delta_seconds</span><span>();
</span><span> }
</span><span>}
</span></code></pre>
<p>And this works. This would be the most naive way to do integration, known as Euler. Specifically Symplectic Euler (because we are figuring out our velocity, then applying it instead of the other way around). You will see objects orbit each other, but eventually all your orbits will degrade. We can figure this out by calculating the total energy (potential + kinetic) of the orbital system and plotting it over time. And the greater the time step, (how much time delta elapsed between each update) the worse the drift gets. </p>
<p>To understand why timestep size matters so much with integration, let's use a simpler analogy.</p>
<p>Imagine simulating a car. And lets say you have a system where you only update the velocity every second. Cars initially accelerate very fast, but their acceleration slows down as air resistance, gearing limitations, and power vs force relationship come into play. Now lets imagine you have two cars and you are going to have a drag race. Each car has the same acceleration curve, but one car's velocity will update every second, and the second car's velocity will update every two seconds. Which car will win? The second car, by a pretty non-trivial amount<sup class="footnote-reference"><a href="#1">1</a></sup>:</p>
<img src="../../images/gravity/drag_race.png" alt="Plot of the position of both cars, showing car B ahead" />
<p>And this is probably intuitive. The car that is calculating its velocity every 2 seconds is accelerating faster for longer. The calculations for its current acceleration aren't wrong, they just aren't happening in realtime.</p>
<p>But, we can make our approximation a bit better. Lets try out a different integration strategy, known as leapfrog<sup class="footnote-reference"><a href="#2">2</a></sup>:</p>
<img src="../../images/gravity/drag_race_2.png" alt="Plot of the position of all 4 cars, showing the two leapfrog integrated cars much closer to reality" />
<p>It doesn't completely solve this whole time step and integration accuracy issue. It is a bit more accurate though. And it does entirely solve some other problems.</p>
<p>The main benefit of leapfrog in the context of Gravitational approximations is that the integration errors don't accumulate like they do with Euler. You can see this on the graph by comparing Euler at dt=1s (blue dot) to Leapfrog at dt=1s. Even though they should be more or less identical, the Leapfrog integration is more conservative. You can see this more at dt=2s. And this effect would be greatly exaggerated at time steps that are an order of magnitude above that. It would also get worse the more steps you simulate.</p>
<p>Another benefit: Leapfrog is <strong>reversible</strong>. Negate time and apply the same steps in reverse, and you'll get right back to where you started. This would be important if I were insane enough to implement this in a multiplayer game with rollback netcode. (I'm not, don't worry.)</p>
<h3 id="implementing-leapfrog-for-gravity">Implementing Leapfrog for Gravity</h3>
<p>Leapfrog integration works in two phases:</p>
<ol>
<li><strong>Kick</strong>: Update velocities using current positions: <code>v(t+dt/2) = v(t-dt/2) + a(t) * dt</code></li>
<li><strong>Drift</strong>: Update positions using new velocities: <code>x(t+dt) = x(t) + v(t+dt/2) * dt</code></li>
</ol>
<p>The key is that positions and velocities are evaluated at <strong>staggered time points</strong>: positions at integer timesteps, velocities at half-timesteps. This leapfrogging is why it can conserve energy on average.</p>
<p>The implementation is quite involved (took me a while to get the math right, and I'm still not 100% sure there aren't bugs). The high-level flow:</p>
<ol>
<li><strong>Snapshot positions</strong> - Capture all body positions at time <code>t</code> for consistent force calculations</li>
<li><strong>Initialize on first run</strong> - Bootstrap velocities to half-timestep: <code>v(-dt/2) = v(0) - a(0) * dt/2</code></li>
<li><strong>Kick phase</strong> - Update all velocities from <code>v(t-dt/2)</code> to <code>v(t+dt/2)</code> using forces at <code>x(t)</code></li>
<li><strong>Drift phase</strong> - Update all positions from <code>x(t)</code> to <code>x(t+dt)</code> using new velocities</li>
</ol>
<p>The tricky part is the initialization step: Leapfrog needs velocities at half-timesteps, so on the first frame we do a half-step <em>backwards</em> to get <code>v(-dt/2)</code> from the initial <code>v(0)</code>.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">/// Trait for integration methods
</span><span style="color:#b48ead;">pub trait </span><span>Integrator {
</span><span> </span><span style="color:#65737e;">/// Perform a single integration step
</span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">integrate_step</span><span>(
</span><span> &</span><span style="color:#bf616a;">self</span><span>,
</span><span> </span><span style="color:#bf616a;">bodies</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Query<(Entity, &</span><span style="color:#b48ead;">mut</span><span> CelestialBody, &</span><span style="color:#b48ead;">mut</span><span> Transform, &GlobalTransform)>,
</span><span> </span><span style="color:#bf616a;">masses</span><span>: &Query<(Entity, &Mass, &GlobalTransform)>,
</span><span> </span><span style="color:#bf616a;">pinned_bodies</span><span>: &Query<Entity, With<Pinned>>,
</span><span> </span><span style="color:#bf616a;">config</span><span>: &GravityConfig,
</span><span> </span><span style="color:#bf616a;">dt</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> );
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>Integrator </span><span style="color:#b48ead;">for </span><span>Leapfrog {
</span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">integrate_step</span><span>(
</span><span> &</span><span style="color:#bf616a;">self</span><span>,
</span><span> </span><span style="color:#bf616a;">bodies</span><span>: &</span><span style="color:#b48ead;">mut </span><span>Query<(Entity, &</span><span style="color:#b48ead;">mut</span><span> CelestialBody, &</span><span style="color:#b48ead;">mut</span><span> Transform, &GlobalTransform)>,
</span><span> </span><span style="color:#bf616a;">masses</span><span>: &Query<(Entity, &Mass, &GlobalTransform)>,
</span><span> </span><span style="color:#bf616a;">pinned_bodies</span><span>: &Query<Entity, With<Pinned>>,
</span><span> </span><span style="color:#bf616a;">config</span><span>: &GravityConfig,
</span><span> </span><span style="color:#bf616a;">dt</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> ) {
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span> </span><span style="color:#65737e;">// LEAPFROG INTEGRATION ALGORITHM
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// The Leapfrog integrator is a second-order symplectic integration method that
</span><span> </span><span style="color:#65737e;">// preserves phase space volume and conserves energy over long time periods.
</span><span> </span><span style="color:#65737e;">// Unlike Euler or RK4, it maintains time-reversibility and stability.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// Position and velocity are evaluated at staggered time points:
</span><span> </span><span style="color:#65737e;">// - Positions x are evaluated at integer timesteps: t, t+dt, t+2dt, ...
</span><span> </span><span style="color:#65737e;">// - Velocities v are evaluated at half-timesteps: t-dt/2, t+dt/2, t+3dt/2, ...
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// This "leapfrogging" of position and velocity gives the method its name and
</span><span> </span><span style="color:#65737e;">// its superior energy conservation properties.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// ALGORITHM (Kick-Drift form):
</span><span> </span><span style="color:#65737e;">// 1. KICK: v(t+dt/2) = v(t-dt/2) + a(t) * dt
</span><span> </span><span style="color:#65737e;">// 2. DRIFT: x(t+dt) = x(t) + v(t+dt/2) * dt
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// Where:
</span><span> </span><span style="color:#65737e;">// - a(t) is acceleration computed from gravitational forces at position x(t)
</span><span> </span><span style="color:#65737e;">// - The "kick" updates velocity using the full timestep dt
</span><span> </span><span style="color:#65737e;">// - The "drift" updates position using the newly updated velocity
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span>
</span><span> </span><span style="color:#65737e;">// ───────────────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">// STEP 0: SNAPSHOT CURRENT POSITIONS
</span><span> </span><span style="color:#65737e;">// ───────────────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// We need to compute accelerations at time t using positions x(t).
</span><span> </span><span style="color:#65737e;">// During the kick phase, we'll be mutably iterating over bodies to update
</span><span> </span><span style="color:#65737e;">// velocities.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// This ensures all bodies use consistent, simultaneous positions when computing
</span><span> </span><span style="color:#65737e;">// gravitational forces, preventing inconsistencies in the force calculation.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#b48ead;">let</span><span> body_positions: Vec<(Entity, Vec3)> = bodies
</span><span> .</span><span style="color:#96b5b4;">iter</span><span>()
</span><span> .</span><span style="color:#96b5b4;">map</span><span>(|(</span><span style="color:#bf616a;">entity</span><span>, _, </span><span style="color:#bf616a;">transform</span><span>, _)| (entity, transform.translation))
</span><span> .</span><span style="color:#96b5b4;">collect</span><span>();
</span><span>
</span><span> </span><span style="color:#65737e;">// ───────────────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">// INITIALIZATION PHASE (First Call Only)
</span><span> </span><span style="color:#65737e;">// ───────────────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// When the simulation starts, we have:
</span><span> </span><span style="color:#65737e;">// - Positions at time t=0: x(0)
</span><span> </span><span style="color:#65737e;">// - Velocities at time t=0: v(0)
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// Leapfrog requires velocities at half-timesteps. Specifically, to start
</span><span> </span><span style="color:#65737e;">// the first full integration step, we need v(-dt/2) not v(0).
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// So, On the very first call, we apply a "half-step backwards kick" to convert
</span><span> </span><span style="color:#65737e;">// v(0) into v(-dt/2). This initialization ensures the staggered time grid
</span><span> </span><span style="color:#65737e;">// is properly established.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// After initialization:
</span><span> </span><span style="color:#65737e;">// - Positions remain at x(0)
</span><span> </span><span style="color:#65737e;">// - Velocities are now at v(-dt/2)
</span><span> </span><span style="color:#65737e;">// - Future steps will maintain this half-timestep offset
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// We return immediately after initialization to avoid doing a full integration
</span><span> </span><span style="color:#65737e;">// step in the same call. This ensures the user sees the correct initial state
</span><span> </span><span style="color:#65737e;">// and the first full step happens on the next call to integrate_step().
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// SAFETY: We access mutable static LEAPFROG_FIRST_RUN to track initialization state.
</span><span> </span><span style="color:#65737e;">// This is safe because Bevy's schedule runs systems sequentially in a single thread,
</span><span> </span><span style="color:#65737e;">// ensuring no concurrent access. The flag is only modified here and in reset_integrator_state().
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#b48ead;">unsafe </span><span>{
</span><span> </span><span style="color:#b48ead;">if </span><span style="color:#d08770;">LEAPFROG_FIRST_RUN </span><span>{
</span><span> </span><span style="color:#65737e;">// ─────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">// INITIALIZATION LOOP: Compute v(-dt/2) from v(0)
</span><span> </span><span style="color:#65737e;">// ─────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#b48ead;">for </span><span>(entity, </span><span style="color:#b48ead;">mut</span><span> body, _transform, _global_transform) in bodies.</span><span style="color:#96b5b4;">iter_mut</span><span>() {
</span><span> </span><span style="color:#65737e;">// Pinned bodies never move; enforce zero velocity
</span><span> </span><span style="color:#b48ead;">if</span><span> pinned_bodies.</span><span style="color:#96b5b4;">contains</span><span>(entity) {
</span><span> body.velocity = Vec3::</span><span style="color:#d08770;">ZERO</span><span>;
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// Retrieve this body's mass
</span><span> </span><span style="color:#b48ead;">let</span><span> our_mass = </span><span style="color:#b48ead;">match</span><span> masses
</span><span> .</span><span style="color:#96b5b4;">iter</span><span>()
</span><span> .</span><span style="color:#96b5b4;">filter</span><span>(|(</span><span style="color:#bf616a;">iter_entity</span><span>, ..)| iter_entity == &entity)
</span><span> .</span><span style="color:#96b5b4;">map</span><span>(|(</span><span style="color:#bf616a;">_iter_entity</span><span>, </span><span style="color:#bf616a;">mass</span><span>, </span><span style="color:#bf616a;">_transform</span><span>)| mass)
</span><span> .collect::<Vec<&Mass>>()
</span><span> .</span><span style="color:#96b5b4;">pop</span><span>()
</span><span> {
</span><span> Some(mass) => mass,
</span><span> None => &Mass::default(),
</span><span> };
</span><span>
</span><span> </span><span style="color:#65737e;">// Retrieve this body's position from the snapshot
</span><span> </span><span style="color:#b48ead;">let</span><span> our_position = body_positions.</span><span style="color:#96b5b4;">iter</span><span>()
</span><span> .</span><span style="color:#96b5b4;">find</span><span>(|(</span><span style="color:#bf616a;">e</span><span>, _)| e == &entity)
</span><span> .</span><span style="color:#96b5b4;">map</span><span>(|(_, </span><span style="color:#bf616a;">pos</span><span>)| *pos)
</span><span> .</span><span style="color:#96b5b4;">unwrap_or</span><span>(Vec3::</span><span style="color:#d08770;">ZERO</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Compute acceleration from all other bodies at time t=0
</span><span> </span><span style="color:#65737e;">// and apply a NEGATIVE half-timestep to move velocity backwards in time
</span><span> </span><span style="color:#b48ead;">for </span><span>(other_entity, other_mass, _other_transform) in masses.</span><span style="color:#96b5b4;">iter</span><span>() {
</span><span> </span><span style="color:#b48ead;">if</span><span> entity != other_entity {
</span><span> </span><span style="color:#b48ead;">let</span><span> other_position = body_positions.</span><span style="color:#96b5b4;">iter</span><span>()
</span><span> .</span><span style="color:#96b5b4;">find</span><span>(|(</span><span style="color:#bf616a;">e</span><span>, _)| e == &other_entity)
</span><span> .</span><span style="color:#96b5b4;">map</span><span>(|(_, </span><span style="color:#bf616a;">pos</span><span>)| *pos)
</span><span> .</span><span style="color:#96b5b4;">unwrap_or</span><span>(Vec3::</span><span style="color:#d08770;">ZERO</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Apply: v(-dt/2) = v(0) + a(0) * (-dt/2)
</span><span> </span><span style="color:#65737e;">// The negative timestep is critical for backwards integration
</span><span> body.velocity += </span><span style="color:#96b5b4;">calculate_velocity_impact_between_positions</span><span>(
</span><span> config,
</span><span> our_position,
</span><span> our_mass,
</span><span> other_position,
</span><span> other_mass,
</span><span> -dt * </span><span style="color:#d08770;">0.5</span><span>, </span><span style="color:#65737e;">// NEGATIVE half-timestep
</span><span> );
</span><span> }
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// Mark initialization complete; future calls will execute normal steps
</span><span> </span><span style="color:#d08770;">LEAPFROG_FIRST_RUN </span><span>= </span><span style="color:#d08770;">false</span><span>;
</span><span>
</span><span> </span><span style="color:#65737e;">// Here, we return without performing a full integration step.
</span><span> </span><span style="color:#65737e;">// At this point we have:
</span><span> </span><span style="color:#65737e;">// - x(0) (unchanged positions)
</span><span> </span><span style="color:#65737e;">// - v(-dt/2) (initialized velocities)
</span><span> </span><span style="color:#65737e;">// The next call will perform the first full kick-drift cycle.
</span><span> </span><span style="color:#b48ead;">return</span><span>;
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span> </span><span style="color:#65737e;">// MAIN INTEGRATION STEP (Kick-Drift)
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// At entry to this section, we have:
</span><span> </span><span style="color:#65737e;">// - Position at full timestep
</span><span> </span><span style="color:#65737e;">// - Velocity at half-timestep
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// We will produce:
</span><span> </span><span style="color:#65737e;">// - Position at next full timestep
</span><span> </span><span style="color:#65737e;">// - Velocity at next half-timestep
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span>
</span><span> </span><span style="color:#65737e;">// ───────────────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">// PHASE 1: KICK
</span><span> </span><span style="color:#65737e;">// ───────────────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// Update velocities from v(t - dt/2) to v(t + dt/2)
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// WHERE:
</span><span> </span><span style="color:#65737e;">// - a(t) is acceleration computed from gravitational forces at x(t)
</span><span> </span><span style="color:#65737e;">// - We use the FULL timestep dt (not dt/2) because we're spanning from
</span><span> </span><span style="color:#65737e;">// (t - dt/2) to (t + dt/2), which is a full timestep interval
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// We MUST compute velocities before updating positions because:
</span><span> </span><span style="color:#65737e;">// 1. Acceleration a(t) depends on positions x(t)
</span><span> </span><span style="color:#65737e;">// 2. If we updated positions first, we'd have x(t+dt), invalidating a(t)
</span><span> </span><span style="color:#65737e;">// 3. The kick-drift order preserves the algorithm's symplectic structure
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// PHYSICAL INTERPRETATION:
</span><span> </span><span style="color:#65737e;">// We're applying the gravitational force at the current positions to
</span><span> </span><span style="color:#65737e;">// "kick" the velocities forward by a full timestep. This represents
</span><span> </span><span style="color:#65737e;">// the momentum change due to forces acting over the interval.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#b48ead;">for </span><span>(entity, </span><span style="color:#b48ead;">mut</span><span> body, _transform, _global_transform) in bodies.</span><span style="color:#96b5b4;">iter_mut</span><span>() {
</span><span> </span><span style="color:#65737e;">// Pinned bodies are immobile; enforce zero velocity
</span><span> </span><span style="color:#b48ead;">if</span><span> pinned_bodies.</span><span style="color:#96b5b4;">contains</span><span>(entity) {
</span><span> body.velocity = Vec3::</span><span style="color:#d08770;">ZERO</span><span>;
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// Retrieve this body's mass
</span><span> </span><span style="color:#b48ead;">let</span><span> our_mass = </span><span style="color:#b48ead;">match</span><span> masses
</span><span> .</span><span style="color:#96b5b4;">iter</span><span>()
</span><span> .</span><span style="color:#96b5b4;">filter</span><span>(|(</span><span style="color:#bf616a;">iter_entity</span><span>, ..)| iter_entity == &entity)
</span><span> .</span><span style="color:#96b5b4;">map</span><span>(|(</span><span style="color:#bf616a;">_iter_entity</span><span>, </span><span style="color:#bf616a;">mass</span><span>, </span><span style="color:#bf616a;">_transform</span><span>)| mass)
</span><span> .collect::<Vec<&Mass>>()
</span><span> .</span><span style="color:#96b5b4;">pop</span><span>()
</span><span> {
</span><span> Some(mass) => mass,
</span><span> None => &Mass::default(),
</span><span> };
</span><span>
</span><span> </span><span style="color:#65737e;">// Retrieve this body's position from the snapshot at time t
</span><span> </span><span style="color:#b48ead;">let</span><span> our_position = body_positions.</span><span style="color:#96b5b4;">iter</span><span>()
</span><span> .</span><span style="color:#96b5b4;">find</span><span>(|(</span><span style="color:#bf616a;">e</span><span>, _)| e == &entity)
</span><span> .</span><span style="color:#96b5b4;">map</span><span>(|(_, </span><span style="color:#bf616a;">pos</span><span>)| *pos)
</span><span> .</span><span style="color:#96b5b4;">unwrap_or</span><span>(Vec3::</span><span style="color:#d08770;">ZERO</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Accumulate gravitational acceleration from all other bodies
</span><span> </span><span style="color:#65737e;">// using positions at time t (from the snapshot)
</span><span> </span><span style="color:#b48ead;">for </span><span>(other_entity, other_mass, _other_transform) in masses.</span><span style="color:#96b5b4;">iter</span><span>() {
</span><span> </span><span style="color:#b48ead;">if</span><span> entity != other_entity {
</span><span> </span><span style="color:#b48ead;">let</span><span> other_position = body_positions.</span><span style="color:#96b5b4;">iter</span><span>()
</span><span> .</span><span style="color:#96b5b4;">find</span><span>(|(</span><span style="color:#bf616a;">e</span><span>, _)| e == &other_entity)
</span><span> .</span><span style="color:#96b5b4;">map</span><span>(|(_, </span><span style="color:#bf616a;">pos</span><span>)| *pos)
</span><span> .</span><span style="color:#96b5b4;">unwrap_or</span><span>(Vec3::</span><span style="color:#d08770;">ZERO</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Apply: v(t+dt/2) = v(t-dt/2) + a(t) * dt
</span><span> </span><span style="color:#65737e;">// This function computes Δv = a * dt from gravitational force
</span><span> body.velocity += </span><span style="color:#96b5b4;">calculate_velocity_impact_between_positions</span><span>(
</span><span> config,
</span><span> our_position,
</span><span> our_mass,
</span><span> other_position,
</span><span> other_mass,
</span><span> dt, </span><span style="color:#65737e;">// FULL timestep (not dt/2!)
</span><span> );
</span><span> }
</span><span> }
</span><span> }
</span><span> </span><span style="color:#65737e;">// At this point:
</span><span> </span><span style="color:#65737e;">// - Positions are still at x(t)
</span><span> </span><span style="color:#65737e;">// - Velocities are now at v(t + dt/2)
</span><span>
</span><span> </span><span style="color:#65737e;">// ───────────────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">// PHASE 2: DRIFT
</span><span> </span><span style="color:#65737e;">// ───────────────────────────────────────────────────────────────────────────────
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// We use v(t + dt/2), which is the velocity at the MIDPOINT of the
</span><span> </span><span style="color:#65737e;">// time interval
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// Using the midpoint velocity rather than v(t) or v(t+dt) is what gives
</span><span> </span><span style="color:#65737e;">// Leapfrog its second-order accuracy and symplectic properties.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// With the updated velocities from the kick phase, we "drift" the positions
</span><span> </span><span style="color:#65737e;">// forward. This represents ballistic motion with the newly computed momentum.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// We must update all velocities before updating any positions to maintain
</span><span> </span><span style="color:#65737e;">// the symplectic structure. If we interleaved kick and drift for each body,
</span><span> </span><span style="color:#65737e;">// we'd break time-reversibility and lose energy conservation.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#b48ead;">for </span><span>(entity, body, </span><span style="color:#b48ead;">mut</span><span> transform, _global_transform) in bodies.</span><span style="color:#96b5b4;">iter_mut</span><span>() {
</span><span> </span><span style="color:#65737e;">// Pinned bodies don't move; skip position update
</span><span> </span><span style="color:#b48ead;">if</span><span> pinned_bodies.</span><span style="color:#96b5b4;">contains</span><span>(entity) {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// Apply: x(t+dt) = x(t) + v(t+dt/2) * dt
</span><span> transform.translation += body.velocity * dt;
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span> </span><span style="color:#65737e;">// POST-STEP STATE
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// After this integration step, the system state is:
</span><span> </span><span style="color:#65737e;">// - Positions are at x(t + dt) [full timestep]
</span><span> </span><span style="color:#65737e;">// - Velocities are at v(t + dt/2) [half-timestep]
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// Note that v(t + dt/2) = v((t+dt) - dt/2), so the half-timestep offset
</span><span> </span><span style="color:#65737e;">// is preserved for the next iteration.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// NEXT ITERATION:
</span><span> </span><span style="color:#65737e;">// The next call will treat the current state as:
</span><span> </span><span style="color:#65737e;">// x(t) ← x(t + dt)
</span><span> </span><span style="color:#65737e;">// v(t - dt/2) ← v(t + dt/2)
</span><span> </span><span style="color:#65737e;">// and perform another kick-drift cycle.
</span><span> </span><span style="color:#65737e;">//
</span><span> </span><span style="color:#65737e;">// ═══════════════════════════════════════════════════════════════════════════════
</span><span> }
</span><span>}
</span></code></pre>
<h3 id="energy-conservation-results">Energy Conservation Results</h3>
<p>The proof is in the numbers. Running 100,000 simulation steps at 60Hz shows the difference. Pay particular attention to the <strong>Energy drift</strong> line in each result:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Simulation configuration:
</span><span> Steps: 100000
</span><span> Timestep: 1/60 second (60Hz physics update)
</span><span> Total simulated time: 1666.7 seconds (27.78 minutes)
</span><span>
</span><span>Running simulation with Symplectic Euler integrator (1x)...
</span><span> Running 100000 updates at 1x speed (1 physics steps per update)...
</span><span> Physics step 10000: Energy = -8.166741e-7, Drift = +1.250555e-12 (+0.0002%)
</span><span> Physics step 20000: Energy = -8.166735e-7, Drift = +1.762146e-12 (+0.0002%)
</span><span> Physics step 30000: Energy = -8.166739e-7, Drift = +1.364242e-12 (+0.0002%)
</span><span> Physics step 40000: Energy = -8.166736e-7, Drift = +1.705303e-12 (+0.0002%)
</span><span> Physics step 50000: Energy = -8.166731e-7, Drift = +2.216893e-12 (+0.0003%)
</span><span> Physics step 60000: Energy = -8.166733e-7, Drift = +1.989520e-12 (+0.0002%)
</span><span> Physics step 70000: Energy = -8.166747e-7, Drift = +5.684342e-13 (+0.0001%)
</span><span> Physics step 80000: Energy = -8.166739e-7, Drift = +1.364242e-12 (+0.0002%)
</span><span> Physics step 90000: Energy = -8.166728e-7, Drift = +2.501110e-12 (+0.0003%)
</span><span> Physics step 100000: Energy = -8.166725e-7, Drift = +2.842171e-12 (+0.0003%)
</span><span>
</span><span>======================================================================
</span><span>Symplectic Euler (1x speed) - Energy Conservation Statistics
</span><span>======================================================================
</span><span>Total physics steps: 100000
</span><span>Energy samples: 100000
</span><span>Initial energy: -8.166753e-7
</span><span>Final energy: -8.166722e-7
</span><span>Energy drift: 3.126388e-12 (+0.0004%)
</span><span>Energy range: 4.376943e-12 (0.0005%)
</span><span>Min energy: -8.166761e-7
</span><span>Max energy: -8.166717e-7
</span><span>Std deviation: 5.339739e-12 (0.0007%)
</span><span>======================================================================
</span><span>
</span><span>
</span><span>Running simulation with Leapfrog integrator (1x)...
</span><span> Running 100000 updates at 1x speed (1 physics steps per update)...
</span><span> Physics step 10000: Energy = -8.166750e-7, Drift = +2.842171e-13 (+0.0000%)
</span><span> Physics step 20000: Energy = -8.166753e-7, Drift = +0.000000e0 (+0.0000%)
</span><span> Physics step 30000: Energy = -8.166757e-7, Drift = -3.410605e-13 (-0.0000%)
</span><span> Physics step 40000: Energy = -8.166762e-7, Drift = -9.094947e-13 (-0.0001%)
</span><span> Physics step 50000: Energy = -8.166758e-7, Drift = -4.547474e-13 (-0.0001%)
</span><span> Physics step 60000: Energy = -8.166767e-7, Drift = -1.364242e-12 (-0.0002%)
</span><span> Physics step 70000: Energy = -8.166754e-7, Drift = -1.136868e-13 (-0.0000%)
</span><span> Physics step 80000: Energy = -8.166760e-7, Drift = -7.389644e-13 (-0.0001%)
</span><span> Physics step 90000: Energy = -8.166757e-7, Drift = -3.979039e-13 (-0.0000%)
</span><span> Physics step 100000: Energy = -8.166747e-7, Drift = +5.684342e-13 (+0.0001%)
</span><span>
</span><span>======================================================================
</span><span>Leapfrog (1x speed) - Energy Conservation Statistics
</span><span>======================================================================
</span><span>Total physics steps: 100000
</span><span>Energy samples: 100000
</span><span>Initial energy: -8.166753e-7
</span><span>Final energy: -8.166744e-7
</span><span>Energy drift: 9.094947e-13 (+0.0001%)
</span><span>Energy range: 3.922196e-12 (0.0005%)
</span><span>Min energy: -8.166778e-7
</span><span>Max energy: -8.166738e-7
</span><span>Std deviation: 5.991950e-12 (0.0007%)
</span><span>======================================================================
</span></code></pre>
<p><strong>Symplectic Euler</strong>: Energy drift of <strong>+0.0004%</strong> after 100,000 steps
<strong>Leapfrog</strong>: Energy drift of <strong>+0.0001%</strong> after 100,000 steps</p>
<p>Leapfrog conserves energy significantly better. And this is at 60Hz, the difference becomes dramatically larger with less frequent timesteps. At 10Hz or 1Hz (common for games that want to speed up time), Euler integration would show visible orbital decay while Leapfrog remains stable.</p>
<h2 id="challenge-3-time-acceleration">Challenge 3: Time Acceleration</h2>
<p>Watching orbits in real-time is pretty boring. </p>
<p>But speeding up the simulation is not trivial. Normally if you wanted to do time acceleration in Bevy you would just do something like this:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">adjust_time_scale</span><span>(</span><span style="color:#bf616a;">keyboard</span><span>: Res<ButtonInput<KeyCode>>, </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">time</span><span>: ResMut<Time<Virtual>>) {
</span><span> </span><span style="color:#b48ead;">if</span><span> keyboard.</span><span style="color:#96b5b4;">just_pressed</span><span>(KeyCode::ArrowUp) {
</span><span> time.</span><span style="color:#96b5b4;">set_relative_speed</span><span>(time.</span><span style="color:#96b5b4;">relative_speed</span><span>() * </span><span style="color:#d08770;">2.0</span><span>);
</span><span> }
</span><span> </span><span style="color:#b48ead;">if</span><span> keyboard.</span><span style="color:#96b5b4;">just_pressed</span><span>(KeyCode::ArrowDown) {
</span><span> time.</span><span style="color:#96b5b4;">set_relative_speed</span><span>(time.</span><span style="color:#96b5b4;">relative_speed</span><span>() * </span><span style="color:#d08770;">0.5</span><span>);
</span><span> }
</span><span> </span><span style="color:#b48ead;">if</span><span> keyboard.</span><span style="color:#96b5b4;">just_pressed</span><span>(KeyCode::Space) {
</span><span> time.</span><span style="color:#96b5b4;">set_relative_speed</span><span>(</span><span style="color:#d08770;">1.0</span><span>);
</span><span> }
</span><span>}
</span></code></pre>
<p>But this doesn't work for fixed-timestep gravity simulation. Here's why: if you set <code>relative_speed</code> to 2.0, Bevy's <code>FixedUpdate</code> still runs at 60Hz, but each step now represents twice as much time. Our gravity calculations would still run 60 times per second, but we'd be applying forces as if much more time had passed. This is the same problem as the car example: larger effective timesteps mean larger integration errors and unstable orbits.</p>
<p>So to do that, we need to accumulate simulation frames when we accelerate time, and skip them when we decelerate time. </p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">/// Resource for controlling simulation time scaling.
</span><span style="color:#65737e;">///
</span><span style="color:#65737e;">/// Supports both time acceleration (for interplanetary travel) and deceleration
</span><span style="color:#65737e;">/// (for precision maneuvers) while maintaining physics accuracy through multi-stepping.
</span><span>#[</span><span style="color:#bf616a;">derive</span><span>(Resource, Debug, Clone)]
</span><span style="color:#b48ead;">pub struct </span><span>TimeScale {
</span><span> </span><span style="color:#65737e;">/// Current time multiplier (1.0 = normal speed, 2.0 = 2x speed, 0.5 = half speed)
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">scale</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#65737e;">/// Whether the simulation is currently paused
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">paused</span><span>: </span><span style="color:#b48ead;">bool</span><span>,
</span><span> </span><span style="color:#65737e;">/// Accumulated fractional steps for sub-1x speeds
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">step_accumulator</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#65737e;">/// Maximum allowed time scale to prevent performance issues
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">max_scale</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#65737e;">/// Minimum allowed time scale
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">min_scale</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>Default </span><span style="color:#b48ead;">for </span><span>TimeScale {
</span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">default</span><span>() -> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> scale: </span><span style="color:#d08770;">1.0</span><span>,
</span><span> paused: </span><span style="color:#d08770;">false</span><span>,
</span><span> step_accumulator: </span><span style="color:#d08770;">0.0</span><span>,
</span><span> max_scale: </span><span style="color:#d08770;">1000.0</span><span>,
</span><span> min_scale: </span><span style="color:#d08770;">0.1</span><span>,
</span><span> }
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>TimeScale {
</span><span> </span><span style="color:#65737e;">/// Create a new TimeScale with custom limits
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">new</span><span>(</span><span style="color:#bf616a;">min_scale</span><span>: </span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#bf616a;">max_scale</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> min_scale,
</span><span> max_scale,
</span><span> ..Default::default()
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Set the time scale, clamping to valid range
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">set_scale</span><span>(&</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">scale</span><span>: </span><span style="color:#b48ead;">f32</span><span>) {
</span><span> </span><span style="color:#bf616a;">self</span><span>.scale = scale.</span><span style="color:#96b5b4;">clamp</span><span>(</span><span style="color:#bf616a;">self</span><span>.min_scale, </span><span style="color:#bf616a;">self</span><span>.max_scale);
</span><span> </span><span style="color:#bf616a;">self</span><span>.step_accumulator = </span><span style="color:#d08770;">0.0</span><span>; </span><span style="color:#65737e;">// Reset accumulator when scale changes
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/// Calculate how many physics steps to run this frame
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">calculate_physics_steps</span><span>(&</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">self</span><span>) -> </span><span style="color:#b48ead;">u32 </span><span>{
</span><span> </span><span style="color:#b48ead;">let</span><span> effective_scale = </span><span style="color:#bf616a;">self</span><span>.</span><span style="color:#96b5b4;">effective_scale</span><span>();
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> effective_scale <= </span><span style="color:#d08770;">0.0 </span><span>{
</span><span> </span><span style="color:#b48ead;">return </span><span style="color:#d08770;">0</span><span>; </span><span style="color:#65737e;">// Paused
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> effective_scale >= </span><span style="color:#d08770;">1.0 </span><span>{
</span><span> </span><span style="color:#65737e;">// For speeds >= 1x, run multiple steps per frame
</span><span> effective_scale.</span><span style="color:#96b5b4;">floor</span><span>() as </span><span style="color:#b48ead;">u32
</span><span> } </span><span style="color:#b48ead;">else </span><span>{
</span><span> </span><span style="color:#65737e;">// For speeds < 1x, accumulate fractional steps
</span><span> </span><span style="color:#bf616a;">self</span><span>.step_accumulator += effective_scale;
</span><span> </span><span style="color:#b48ead;">if </span><span style="color:#bf616a;">self</span><span>.step_accumulator >= </span><span style="color:#d08770;">1.0 </span><span>{
</span><span> </span><span style="color:#bf616a;">self</span><span>.step_accumulator -= </span><span style="color:#d08770;">1.0</span><span>;
</span><span> </span><span style="color:#d08770;">1
</span><span> } </span><span style="color:#b48ead;">else </span><span>{
</span><span> </span><span style="color:#d08770;">0
</span><span> }
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<p>And this brings us conveniently back to the actual system that controls all of the physics, where these steps are actually run:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">/// Physics update system that performs gravity integration using the configured integrator
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">update_physics</span><span>(
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">bodies</span><span>: Query<(Entity, &</span><span style="color:#b48ead;">mut</span><span> CelestialBody, &</span><span style="color:#b48ead;">mut</span><span> Transform, &GlobalTransform)>,
</span><span> </span><span style="color:#bf616a;">masses</span><span>: Query<(Entity, &Mass, &GlobalTransform)>,
</span><span> </span><span style="color:#bf616a;">pinned_bodies</span><span>: Query<Entity, With<Pinned>>,
</span><span> </span><span style="color:#bf616a;">time</span><span>: Res<Time<Fixed>>,
</span><span> </span><span style="color:#bf616a;">config</span><span>: Res<GravityConfig>,
</span><span> </span><span style="color:#bf616a;">integrator_config</span><span>: Res<IntegratorConfig>,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">time_scale</span><span>: ResMut<TimeScale>,
</span><span>) {
</span><span> </span><span style="color:#65737e;">// Calculate how many physics steps to run this frame based on time scale
</span><span> </span><span style="color:#b48ead;">let</span><span> steps_to_run = time_scale.</span><span style="color:#96b5b4;">calculate_physics_steps</span><span>();
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> steps_to_run == </span><span style="color:#d08770;">0 </span><span>{
</span><span> </span><span style="color:#b48ead;">return</span><span>; </span><span style="color:#65737e;">// Paused or no steps needed this frame
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// Use fixed timestep for all calculations to maintain determinism
</span><span> </span><span style="color:#b48ead;">let</span><span> dt = time.</span><span style="color:#96b5b4;">delta_secs</span><span>();
</span><span>
</span><span> </span><span style="color:#65737e;">// Run multiple physics steps for time acceleration
</span><span> </span><span style="color:#b48ead;">for</span><span> _step in </span><span style="color:#d08770;">0</span><span>..steps_to_run {
</span><span> integrators::perform_integration_step(
</span><span> integrator_config.method,
</span><span> &</span><span style="color:#b48ead;">mut</span><span> bodies,
</span><span> &masses,
</span><span> &pinned_bodies,
</span><span> &config,
</span><span> dt,
</span><span> );
</span><span> }
</span><span>}
</span></code></pre>
<p>And very importantly, this system <em>must</em> be run at a fixed update:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">impl </span><span>Plugin </span><span style="color:#b48ead;">for </span><span>CelestialBodyPlugin {
</span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">build</span><span>(&</span><span style="color:#bf616a;">self</span><span>, </span><span style="color:#bf616a;">app</span><span>: &</span><span style="color:#b48ead;">mut</span><span> App) {
</span><span> app.</span><span style="color:#96b5b4;">insert_resource</span><span>(</span><span style="color:#bf616a;">self</span><span>.config.</span><span style="color:#96b5b4;">clone</span><span>())
</span><span> .</span><span style="color:#96b5b4;">insert_resource</span><span>(PathPredictionConfig::default())
</span><span> .</span><span style="color:#96b5b4;">insert_resource</span><span>(TimeScale::default())
</span><span> .</span><span style="color:#96b5b4;">insert_resource</span><span>(</span><span style="color:#bf616a;">self</span><span>.integrator_config.</span><span style="color:#96b5b4;">clone</span><span>())
</span><span> .register_type::<CelestialBody>()
</span><span> .register_type::<Mass>()
</span><span> .register_type::<Length>()
</span><span> .register_type::<TimeInterval>()
</span><span> .register_type::<Pinned>()
</span><span> .register_type::<PathPrediction>()
</span><span> .register_type::<IntegratorConfig>()
</span><span> .register_type::<IntegratorType>()
</span><span> .</span><span style="color:#96b5b4;">add_systems</span><span>(FixedUpdate, update_physics) </span><span style="color:#65737e;">// <- Here
</span><span> .</span><span style="color:#96b5b4;">add_systems</span><span>(Update, (
</span><span> path_prediction::predict_paths,
</span><span> path_prediction::update_path_visualizations,
</span><span> path_prediction::cleanup_path_visualizations,
</span><span> ).</span><span style="color:#96b5b4;">chain</span><span>());
</span><span> }
</span><span>}
</span></code></pre>
<p>And that covers the core mechanics.</p>
<h2 id="putting-it-all-together">Putting It All Together</h2>
<p>Here's a minimal example showing how to use the plugin:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">use </span><span>bevy::prelude::*;
</span><span style="color:#b48ead;">use </span><span>bevy_gravity::*;
</span><span>
</span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">main</span><span>() {
</span><span> App::new()
</span><span> </span><span style="color:#65737e;">// Set black background
</span><span> .</span><span style="color:#96b5b4;">insert_resource</span><span>(ClearColor(Color::</span><span style="color:#d08770;">BLACK</span><span>))
</span><span> </span><span style="color:#65737e;">// Add default Bevy plugins with window config
</span><span> .</span><span style="color:#96b5b4;">add_plugins</span><span>(DefaultPlugins.</span><span style="color:#96b5b4;">set</span><span>(WindowPlugin {
</span><span> primary_window: Some(Window {
</span><span> title: "</span><span style="color:#a3be8c;">Minimal Two Body Orbit</span><span>".</span><span style="color:#96b5b4;">into</span><span>(),
</span><span> resolution: (</span><span style="color:#d08770;">1200</span><span>, </span><span style="color:#d08770;">800</span><span>).</span><span style="color:#96b5b4;">into</span><span>(),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> }),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> }))
</span><span> </span><span style="color:#65737e;">// Add gravity simulation plugin with scale settings
</span><span> </span><span style="color:#65737e;">// 1 unit = 1000 km, masses measured in Earth masses
</span><span> .</span><span style="color:#96b5b4;">add_plugins</span><span>(CelestialBodyPlugin {
</span><span> config: GravityConfig {
</span><span> length_scale: Length::from_kilometers(</span><span style="color:#d08770;">1000.0</span><span>),
</span><span> mass_scale: Mass::earth(),
</span><span> },
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> })
</span><span> </span><span style="color:#65737e;">// Configure orbital path prediction
</span><span> </span><span style="color:#65737e;">// This draws the predicted orbit paths for each body
</span><span> .</span><span style="color:#96b5b4;">insert_resource</span><span>(PathPredictionConfig {
</span><span> prediction_steps: </span><span style="color:#d08770;">100000</span><span>,
</span><span> prediction_timestep: TimeInterval::from_seconds(</span><span style="color:#d08770;">1.0 </span><span>/ </span><span style="color:#d08770;">60.0</span><span>),
</span><span> enabled: </span><span style="color:#d08770;">true</span><span>,
</span><span> auto_disable_threshold: None,
</span><span> })
</span><span> </span><span style="color:#65737e;">// Set time acceleration to 10x speed
</span><span> .</span><span style="color:#96b5b4;">insert_resource</span><span>(TimeScale {
</span><span> scale: </span><span style="color:#d08770;">10.0</span><span>,
</span><span> ..Default::default()
</span><span> })
</span><span> </span><span style="color:#65737e;">// Run setup functions on startup
</span><span> .</span><span style="color:#96b5b4;">add_systems</span><span>(Startup, (spawn_camera, spawn_scene))
</span><span> .</span><span style="color:#96b5b4;">run</span><span>();
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">spawn_camera</span><span>(</span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">commands</span><span>: Commands) {
</span><span> </span><span style="color:#65737e;">// Camera positioned above the scene looking down at the orbital plane
</span><span> commands.</span><span style="color:#96b5b4;">spawn</span><span>((
</span><span> Camera3d::default(),
</span><span> Transform::from_xyz(</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">8.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>).</span><span style="color:#96b5b4;">looking_at</span><span>(Vec3::</span><span style="color:#d08770;">ZERO</span><span>, Vec3::Z),
</span><span> ));
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">spawn_scene</span><span>(
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">commands</span><span>: Commands,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">meshes</span><span>: ResMut<Assets<Mesh>>,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">materials</span><span>: ResMut<Assets<StandardMaterial>>,
</span><span> </span><span style="color:#bf616a;">config</span><span>: Res<GravityConfig>,
</span><span>) {
</span><span> </span><span style="color:#65737e;">// Add a light source so we can see the spheres
</span><span> commands.</span><span style="color:#96b5b4;">spawn</span><span>((
</span><span> PointLight {
</span><span> intensity: </span><span style="color:#d08770;">1500.0</span><span>,
</span><span> shadows_enabled: </span><span style="color:#d08770;">true</span><span>,
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> },
</span><span> Transform::from_xyz(</span><span style="color:#d08770;">4.0</span><span>, </span><span style="color:#d08770;">8.0</span><span>, </span><span style="color:#d08770;">4.0</span><span>),
</span><span> ));
</span><span>
</span><span> </span><span style="color:#65737e;">// Create the central body at the origin
</span><span> </span><span style="color:#65737e;">// This is a blue sphere with Earth's mass that stays stationary
</span><span> </span><span style="color:#b48ead;">let</span><span> central_mass = Mass::earth();
</span><span> commands.</span><span style="color:#96b5b4;">spawn</span><span>((
</span><span> Mesh3d(meshes.</span><span style="color:#96b5b4;">add</span><span>(Sphere::new(</span><span style="color:#d08770;">0.4</span><span>))),
</span><span> MeshMaterial3d(materials.</span><span style="color:#96b5b4;">add</span><span>(StandardMaterial {
</span><span> base_color: Color::srgb(</span><span style="color:#d08770;">0.3</span><span>, </span><span style="color:#d08770;">0.7</span><span>, </span><span style="color:#d08770;">1.0</span><span>),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> })),
</span><span> Transform::from_xyz(</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>),
</span><span> central_mass.</span><span style="color:#96b5b4;">clone</span><span>(),
</span><span> CelestialBody {
</span><span> velocity: Vec3::</span><span style="color:#d08770;">ZERO</span><span>,
</span><span> },
</span><span> PathPrediction {
</span><span> display_path: </span><span style="color:#d08770;">true</span><span>,
</span><span> path_color: Color::srgb(</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>),
</span><span> max_display_points: </span><span style="color:#d08770;">1000000</span><span>,
</span><span> },
</span><span> ));
</span><span>
</span><span> </span><span style="color:#65737e;">// Create the orbiting body
</span><span> </span><span style="color:#65737e;">// Position it 1 unit away from center (1000 km in real space)
</span><span> </span><span style="color:#b48ead;">let</span><span> orbital_position = Vec3::new(</span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>);
</span><span> </span><span style="color:#65737e;">// Orbit rotates around the Y axis (so orbit happens in the XZ plane)
</span><span> </span><span style="color:#b48ead;">let</span><span> orbit_axis = Vec3::Y;
</span><span>
</span><span> </span><span style="color:#65737e;">// Calculate the exact velocity needed for a stable circular orbit
</span><span> </span><span style="color:#65737e;">// This takes into account the central mass and distance
</span><span> </span><span style="color:#b48ead;">let</span><span> orbital_velocity = </span><span style="color:#96b5b4;">calculate_orbital_velocity_vector</span><span>(
</span><span> &config,
</span><span> &central_mass,
</span><span> orbital_position,
</span><span> orbit_axis,
</span><span> );
</span><span>
</span><span> </span><span style="color:#65737e;">// Spawn the orbiting body as a gray sphere with the calculated velocity
</span><span> commands.</span><span style="color:#96b5b4;">spawn</span><span>((
</span><span> Mesh3d(meshes.</span><span style="color:#96b5b4;">add</span><span>(Sphere::new(</span><span style="color:#d08770;">0.15</span><span>))),
</span><span> MeshMaterial3d(materials.</span><span style="color:#96b5b4;">add</span><span>(StandardMaterial {
</span><span> base_color: Color::srgb(</span><span style="color:#d08770;">0.8</span><span>, </span><span style="color:#d08770;">0.8</span><span>, </span><span style="color:#d08770;">0.7</span><span>),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> })),
</span><span> Transform::from_translation(orbital_position),
</span><span> Mass::from_kg(1e15),
</span><span> CelestialBody {
</span><span> velocity: orbital_velocity,
</span><span> },
</span><span> PathPrediction {
</span><span> display_path: </span><span style="color:#d08770;">true</span><span>,
</span><span> path_color: Color::srgb(</span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>),
</span><span> max_display_points: </span><span style="color:#d08770;">1000000</span><span>,
</span><span> },
</span><span> ));
</span><span>}
</span></code></pre>
<h2 id="interactive-demo">Interactive Demo</h2>
<p>Check out an interactive demo <a href="../../projects/bevy-gravity/">here</a></p>
<h1 id="future-plans-and-actual-usage">Future Plans, and Actual Usage</h1>
<p>So, is all this practical for a real game? To be honest, probably not.</p>
<p>It's probably not practical to have <em>every single entity</em> participate in the n-body simulations. For one, having to predict the path of a bunch of other objects that are all subject to n-body physics themselves requires subjecting yourself to a lot of brute-force calculations (which is what the current path prediction does). The next time I pick this up, I'll probably create a <code>PinnedOrbit</code> component, where a given entity can find a stable orbit around a <code>Pinned</code> entity, calculate a realistic orbit, and then just follow that path on rails for the rest of eternity. This would make sense for a game involving space craft with realistic orbital physics. In a stable orbital system, the orbits of the planets are not meaningfully going to change, and the mass of all the spacecraft zipping around aren't going to meaningfully affect the trajectory of the planets due to the astronomical mass disparity. Calculating all that out would just be a waste of CPU cycles. </p>
<p>But this does lay the groundwork for having to realistically navigate a solar system with real gravity in some kind of craft, which was the main goal anyways. </p>
<p>Other practical considerations are that even with being able to set the "scale" of the gravity simulation via the config, there are probably still going to be situations where floating point errors arise. But I think this is something that could be solved with things the way they currently are. You might have a gravity system with all planets when the player entity is out in the middle of nowhere, and then transparently switch to a low orbit scale when the player entity is approaching a station for instance. With any game at a scale like this, some kind of <a href="https://netherlands3d.eu/docs/developers/features/floating-origin/">floating origin</a> shenanigans are probably going to be necessary anyway. Extending that concept to the scale of the gravity situation probably makes sense. </p>
<p>So having a bunch of fixed planets in fixed orbits, but subjecting the actual player entity (or entities) to n-body physics is much more reasonable. It's transparent from a gameplay perspective, and much better optimized from a resource perspective.</p>
<p>Check out the repo and examples <a href="https://github.com/graysonhead/bevy-gravity">here</a> if you'd like to give it a spin.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p><a href="https://gist.github.com/graysonhead/224810ebb92fe532444b56a45899d4eb">Python implementation of Euler integration</a></p>
</div>
<div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup>
<p><a href="https://gist.github.com/graysonhead/afe5a20716d910b9ad2e1a47ae47d022">Python implementation of Leapfrog integration</a></p>
</div>
Bevy Procedural Earth Part 3: Visualizations2025-04-23T00:00:00+00:002025-04-23T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/bevy-proc-earth-3/<img src="../../images/earth_banner_3.png" alt="A view of Greenland with Exaggerated Elevation" />
<p>The main goal of this project for me wasn't primarily about game development, but rather just to have a re-usable way to do global visualizations. One of the most interesting things to visualize on a planet is elevation. So in this tutorial we will look at actually displacing vertices in order to give our sphere some depth. This approach utilizes the high vertex count we generated in previous examples to create meaningful terrain detail.</p>
<p>In addition, I'll give an example of how you might use this program to visualize non-elevation data.</p>
<h1 id="elevation-dataset">Elevation Dataset</h1>
<p>I'm using the <a href="https://www.ncei.noaa.gov/maps/grid-extract/">ETOPO surface elevation models</a> from NOAA in GeoTIFF format at a 60 arcsecond resolution. 60 arcseconds (approximately 1.1 miles at Earth's average surface elevation) is perfectly adequate for what we are doing here. Working with raster files is also nice, because (as discussed in the previous post in this series) we can easily map Latitudes and Longitudes onto the (X, Y) coordinates of our raster using affine transforms (which almost all GIS libraries support).</p>
<h1 id="reading-data-from-the-raster">Reading Data From The Raster</h1>
<p>The core functionality is implemented in this struct::</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub struct </span><span>RasterData {
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">dataset</span><span>: Dataset,
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">transform</span><span>: CoordTransform,
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>RasterData {
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">new</span><span>(</span><span style="color:#bf616a;">path</span><span>: &</span><span style="color:#b48ead;">str</span><span>) -> Result<</span><span style="color:#b48ead;">Self</span><span>, GdalError> {
</span><span> </span><span style="color:#b48ead;">let</span><span> dataset = Dataset::open(path)?;
</span><span> </span><span style="color:#b48ead;">let</span><span> srs = dataset.</span><span style="color:#96b5b4;">spatial_ref</span><span>()?;
</span><span> </span><span style="color:#b48ead;">let</span><span> target_srs = SpatialRef::from_epsg(</span><span style="color:#d08770;">4326</span><span>)?;
</span><span> </span><span style="color:#b48ead;">let</span><span> transform = gdal::spatial_ref::CoordTransform::new(&srs, &target_srs)?;
</span><span> Ok(</span><span style="color:#b48ead;">Self </span><span>{ dataset, transform })
</span><span> }
</span><span> </span><span style="color:#65737e;">// Takes a latitude and longitude in WGS84 coordinates (EPSG:4326) and returns the elevation at that point
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">get_coordinate_height</span><span>(
</span><span> &</span><span style="color:#bf616a;">self</span><span>,
</span><span> </span><span style="color:#bf616a;">latitude</span><span>: </span><span style="color:#b48ead;">f64</span><span>,
</span><span> </span><span style="color:#bf616a;">longitude</span><span>: </span><span style="color:#b48ead;">f64</span><span>,
</span><span> ) -> Result<Option<</span><span style="color:#b48ead;">f64</span><span>>, GdalError> {
</span><span> </span><span style="color:#65737e;">// Copy the input coordinates
</span><span> </span><span style="color:#b48ead;">let </span><span>(lat, lon) = (latitude, longitude);
</span><span>
</span><span> </span><span style="color:#65737e;">// Transform the coordinates from everyone's favorite datum (WGS84) to the raster's native coordinate system
</span><span> </span><span style="color:#bf616a;">self</span><span>.transform
</span><span> .</span><span style="color:#96b5b4;">transform_coords</span><span>(&</span><span style="color:#b48ead;">mut </span><span>[lon], &</span><span style="color:#b48ead;">mut </span><span>[lat], &</span><span style="color:#b48ead;">mut </span><span>[])?;
</span><span>
</span><span> </span><span style="color:#65737e;">// Get the first raster band (usually the only one for elevation data)
</span><span> </span><span style="color:#b48ead;">let</span><span> raster_band = </span><span style="color:#bf616a;">self</span><span>.dataset.</span><span style="color:#96b5b4;">rasterband</span><span>(</span><span style="color:#d08770;">1</span><span>)?;
</span><span>
</span><span> </span><span style="color:#65737e;">// Get the affine transformation parameters that map between pixel/line coordinates and georeferenced coordinates
</span><span> </span><span style="color:#b48ead;">let</span><span> transform = </span><span style="color:#bf616a;">self</span><span>.dataset.</span><span style="color:#96b5b4;">geo_transform</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span>
</span><span> </span><span style="color:#65737e;">// Calculate the pixel (x) and line (y) coordinates in the raster using the affine transform
</span><span> </span><span style="color:#65737e;">// transform[0] = top left x coordinate (origin)
</span><span> </span><span style="color:#65737e;">// transform[1] = pixel width (x resolution)
</span><span> </span><span style="color:#65737e;">// transform[3] = top left y coordinate (origin)
</span><span> </span><span style="color:#65737e;">// transform[5] = pixel height (y resolution, typically negative as y decreases going down)
</span><span> </span><span style="color:#b48ead;">let</span><span> x = (lon - transform[</span><span style="color:#d08770;">0</span><span>]) / transform[</span><span style="color:#d08770;">1</span><span>];
</span><span> </span><span style="color:#b48ead;">let</span><span> y = (lat - transform[</span><span style="color:#d08770;">3</span><span>]) / transform[</span><span style="color:#d08770;">5</span><span>];
</span><span>
</span><span> </span><span style="color:#65737e;">// Read the elevation value at the calculated pixel position
</span><span> </span><span style="color:#65737e;">// - Reads a 1x1 window at position (x,y)
</span><span> </span><span style="color:#65737e;">// - Uses the Average resampling algorithm (which doesn't matter much for a 1x1 window)
</span><span> </span><span style="color:#65737e;">// - Returns the data as f64 (double precision floating point)
</span><span> </span><span style="color:#b48ead;">let mut</span><span> res_buffer = raster_band.read_as::<</span><span style="color:#b48ead;">f64</span><span>>(
</span><span> (x as </span><span style="color:#b48ead;">isize</span><span>, y as </span><span style="color:#b48ead;">isize</span><span>), </span><span style="color:#65737e;">// Pixel position (cast to integer)
</span><span> (</span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">1</span><span>), </span><span style="color:#65737e;">// Window size to read (1x1 pixel)
</span><span> (</span><span style="color:#d08770;">1</span><span>, </span><span style="color:#d08770;">1</span><span>), </span><span style="color:#65737e;">// Output buffer size
</span><span> Some(ResampleAlg::Average),</span><span style="color:#65737e;">// Resampling algorithm
</span><span> )?;
</span><span>
</span><span> </span><span style="color:#65737e;">// Return the elevation value (or None if no data is found)
</span><span> </span><span style="color:#65737e;">// pop() returns and removes the last element from res_buffer.data
</span><span> Ok(res_buffer.data.</span><span style="color:#96b5b4;">pop</span><span>())
</span><span> }
</span><span>}
</span></code></pre>
<p>Here's an example demonstrating how to use this functionality to retrieve elevation data:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">test</span><span>]
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">test_raster_map</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> raster_data =
</span><span> RasterData::new("</span><span style="color:#a3be8c;">assets/Bathymetry/gebco_2023_n47.7905_s39.9243_w25.6311_e42.9895.tif</span><span>")
</span><span> .</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span>
</span><span> </span><span style="color:#65737e;">// Mt Elbrus
</span><span> </span><span style="color:#b48ead;">let</span><span> tgt_latitude = </span><span style="color:#d08770;">43.351851</span><span>;
</span><span> </span><span style="color:#b48ead;">let</span><span> tgt_longitude = </span><span style="color:#d08770;">42.4368771</span><span>;
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> elevation = raster_data
</span><span> .</span><span style="color:#96b5b4;">get_coordinate_height</span><span>(tgt_latitude, tgt_longitude)
</span><span> .</span><span style="color:#96b5b4;">unwrap</span><span>()
</span><span> .</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span>
</span><span> assert_eq!(elevation, </span><span style="color:#d08770;">5392.0</span><span>);
</span><span>}
</span></code></pre>
<p>And that is basically all there is to extracting data from a raster. If you want to be more precise, it might be useful to calculate the average area represented by each vertex and sample a larger size, averaging the result (which you can adjust by changing the window size) or using a different resampling technique. But since our goal isn't to create ultra-accurate maps I think this should be good enough.</p>
<h1 id="moving-pixels">Moving Pixels</h1>
<p>Now, I first want to say, this is an extremely unoptimized naive way to do this. If I had more time to dedicate to this little experiment, I would probably split the rendering of each tile (since we constructed our sphere from multiple distinct Entities) into multiple systems so it can generate the offsets in paralell. But perfect is the enemy of done, so here is something that at least works. </p>
<p>Modifying our pre-existing <code>generate_face</code> function, we can accomplish what we need to do with two additional lines:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">generate_face</span><span>(
</span><span> </span><span style="color:#bf616a;">normal</span><span>: Vec3,
</span><span> </span><span style="color:#bf616a;">resolution</span><span>: </span><span style="color:#b48ead;">u32</span><span>,
</span><span> </span><span style="color:#bf616a;">x_offset</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#bf616a;">y_offset</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#bf616a;">rs</span><span>: &RasterData,
</span><span>) -> Mesh {
</span><span> </span><span style="color:#b48ead;">let</span><span> axis_a = Vec3::new(normal.y, normal.z, normal.x); </span><span style="color:#65737e;">// Horizontal
</span><span> </span><span style="color:#b48ead;">let</span><span> axis_b = axis_a.</span><span style="color:#96b5b4;">cross</span><span>(normal); </span><span style="color:#65737e;">// Vertical
</span><span>
</span><span> </span><span style="color:#65737e;">// Create a vec of verticies and indicies
</span><span> </span><span style="color:#b48ead;">let mut</span><span> verticies: Vec<Vec3> = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> uvs = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> indicies: Vec<</span><span style="color:#b48ead;">u32</span><span>> = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> normals = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> first_longitude = </span><span style="color:#d08770;">0.0</span><span>;
</span><span> </span><span style="color:#b48ead;">for</span><span> y in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">for</span><span> x in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">let</span><span> i = x + y * resolution;
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> percent = Vec2::new(x as </span><span style="color:#b48ead;">f32</span><span>, y as </span><span style="color:#b48ead;">f32</span><span>) / (resolution - </span><span style="color:#d08770;">1</span><span>) as </span><span style="color:#b48ead;">f32</span><span>;
</span><span> </span><span style="color:#b48ead;">let</span><span> point_on_unit_cube =
</span><span> normal + (percent.x - x_offset) * axis_a + (percent.y - y_offset) * axis_b;
</span><span> </span><span style="color:#b48ead;">let</span><span> point_coords: Coordinates = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>().</span><span style="color:#96b5b4;">into</span><span>();
</span><span> </span><span style="color:#b48ead;">let </span><span>(lat, lon) = point_coords.</span><span style="color:#96b5b4;">as_degrees</span><span>();
</span><span> </span><span style="color:#65737e;">// Get the height value at the geographic coordinates
</span><span> </span><span style="color:#b48ead;">let</span><span> height_offset = rs.</span><span style="color:#96b5b4;">get_coordinate_height</span><span>(lat as </span><span style="color:#b48ead;">f64</span><span>, lon as </span><span style="color:#b48ead;">f64</span><span>);
</span><span> </span><span style="color:#65737e;">// Add the elevation to the earth_radius value of the normalized point
</span><span> </span><span style="color:#b48ead;">let</span><span> normalized_point = </span><span style="color:#b48ead;">if let </span><span>Ok(Some(offset)) = height_offset {
</span><span> </span><span style="color:#b48ead;">let</span><span> height = </span><span style="color:#b48ead;">if</span><span> offset > </span><span style="color:#d08770;">0.0 </span><span>{ offset / </span><span style="color:#d08770;">1000.0 </span><span>} </span><span style="color:#b48ead;">else </span><span>{ </span><span style="color:#d08770;">0.0 </span><span>};
</span><span> point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>() * (</span><span style="color:#d08770;">EARTH_RADIUS </span><span>+ (height) as </span><span style="color:#b48ead;">f32</span><span>)
</span><span> } </span><span style="color:#b48ead;">else </span><span>{
</span><span> point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS
</span><span> };
</span><span>
</span><span> verticies.</span><span style="color:#96b5b4;">push</span><span>(normalized_point);
</span><span> </span><span style="color:#b48ead;">let </span><span>(</span><span style="color:#b48ead;">mut</span><span> u, v) = point_coords.</span><span style="color:#96b5b4;">convert_to_uv_mercator</span><span>();
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> y == </span><span style="color:#d08770;">0 </span><span>&& x == </span><span style="color:#d08770;">0 </span><span>{
</span><span> first_longitude = lon;
</span><span> }
</span><span> </span><span style="color:#65737e;">// In the middle latitudes, if we start on a negative longitude but then wind up crossing to a positive longitude, set u to 0.0 to prevent a seam
</span><span> </span><span style="color:#b48ead;">if</span><span> first_longitude < </span><span style="color:#d08770;">0.0 </span><span>&& lon > </span><span style="color:#d08770;">0.0 </span><span>&& lat < </span><span style="color:#d08770;">89.0 </span><span>&& lat > -</span><span style="color:#d08770;">89.0 </span><span>{
</span><span> u = </span><span style="color:#d08770;">0.0</span><span>;
</span><span> }
</span><span> </span><span style="color:#65737e;">// If we are below -40 degrees latitude and the tile starts at 180 degrees, set u to 0.0 to prevent a seam
</span><span> </span><span style="color:#b48ead;">if</span><span> x == </span><span style="color:#d08770;">0 </span><span>&& lon == </span><span style="color:#d08770;">180.0 </span><span>&& lat < -</span><span style="color:#d08770;">40.0 </span><span>{
</span><span> u = </span><span style="color:#d08770;">0.0</span><span>;
</span><span> }
</span><span> uvs.</span><span style="color:#96b5b4;">push</span><span>([u, v]);
</span><span> normals.</span><span style="color:#96b5b4;">push</span><span>(-point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>());
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> x != resolution - </span><span style="color:#d08770;">1 </span><span>&& y != resolution - </span><span style="color:#d08770;">1 </span><span>{
</span><span> </span><span style="color:#65737e;">// First triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Second triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + </span><span style="color:#d08770;">1</span><span>);
</span><span> }
</span><span> }
</span><span> }
</span><span> </span><span style="color:#b48ead;">let</span><span> indicies = mesh::Indices::</span><span style="color:#d08770;">U32</span><span>(indicies);
</span><span> </span><span style="color:#b48ead;">let mut</span><span> mesh = Mesh::new(PrimitiveTopology::TriangleList);
</span><span> mesh.</span><span style="color:#96b5b4;">set_indices</span><span>(Some(indicies));
</span><span> mesh.</span><span style="color:#96b5b4;">insert_attribute</span><span>(Mesh::</span><span style="color:#d08770;">ATTRIBUTE_POSITION</span><span>, verticies);
</span><span> mesh.</span><span style="color:#96b5b4;">insert_attribute</span><span>(Mesh::</span><span style="color:#d08770;">ATTRIBUTE_NORMAL</span><span>, normals);
</span><span> mesh.</span><span style="color:#96b5b4;">insert_attribute</span><span>(Mesh::</span><span style="color:#d08770;">ATTRIBUTE_UV_0</span><span>, uvs);
</span><span> mesh.</span><span style="color:#96b5b4;">generate_tangents</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> mesh
</span><span>}
</span></code></pre>
<p>Specifically, the lines we added:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">// Get the height value at the geographic coordinates
</span><span>lvet height_offset = rs.</span><span style="color:#96b5b4;">get_coordinate_height</span><span>(lat as </span><span style="color:#b48ead;">f64</span><span>, lon as </span><span style="color:#b48ead;">f64</span><span>);
</span><span style="color:#65737e;">// Add the elevation to the earth_radius value of the normalized point
</span><span style="color:#b48ead;">let</span><span> normalized_point = </span><span style="color:#b48ead;">if let </span><span>Ok(Some(offset)) = height_offset {
</span><span> </span><span style="color:#b48ead;">let</span><span> height = </span><span style="color:#b48ead;">if</span><span> offset > </span><span style="color:#d08770;">0.0 </span><span>{ offset / </span><span style="color:#d08770;">1000.0 </span><span>} </span><span style="color:#b48ead;">else </span><span>{ </span><span style="color:#d08770;">0.0 </span><span>};
</span><span> point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>() * (</span><span style="color:#d08770;">EARTH_RADIUS </span><span>+ (height) as </span><span style="color:#b48ead;">f32</span><span>)
</span><span>} </span><span style="color:#b48ead;">else </span><span>{
</span><span> point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS
</span><span>};
</span></code></pre>
<p>As you can see, this is pretty simple since it just builds off of what we already have. You can change the number that divides the offset to adjust how exaggerated the values are. If <code>EARTH_RADIUS</code> is the radius of the earth in meters, a value of <code>1.0</code> should give you a perfectly proportional earth. Of course, you won't move each vertex much at all at that point, and with the number of vertices in the sphere, you wouldn't be able to see anything anyways.</p>
<img src="../../images/bevy_earth_final_elevation.png" alt="An exaggerated elevation map using the above settings" />
<h1 id="visualizing-other-data">Visualizing Other Data</h1>
<p>Assuming you want to visualize using new entities, you can easily do so as long as you have a value you want to base the color/shape/size of the new entity on, and the latitude and longitude of the datapoint by converting the LatLon to a <code>Vec3</code> of a point on the sphere. Here is the struct I use to do all my Geographic/Cartesian coordinate conversions for reference:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">derive</span><span>(Debug)]
</span><span style="color:#b48ead;">pub struct </span><span>Coordinates {
</span><span> </span><span style="color:#65737e;">// Stored internally in radians (because math)
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">latitude</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">longitude</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>From<Vec3> </span><span style="color:#b48ead;">for </span><span>Coordinates {
</span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">from</span><span>(</span><span style="color:#bf616a;">value</span><span>: Vec3) -> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> </span><span style="color:#b48ead;">let</span><span> normalized_point = value.</span><span style="color:#96b5b4;">normalize</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> latitude = normalized_point.y.</span><span style="color:#96b5b4;">asin</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> longitude = normalized_point.x.</span><span style="color:#96b5b4;">atan2</span><span>(normalized_point.z);
</span><span> Coordinates {
</span><span> latitude,
</span><span> longitude,
</span><span> }
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>Coordinates {
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">as_degrees</span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> (</span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#b48ead;">f32</span><span>) {
</span><span> </span><span style="color:#b48ead;">let</span><span> latitude = </span><span style="color:#bf616a;">self</span><span>.latitude * (</span><span style="color:#d08770;">180.0 </span><span>/ </span><span style="color:#d08770;">PI</span><span>);
</span><span> </span><span style="color:#b48ead;">let</span><span> longitude = </span><span style="color:#bf616a;">self</span><span>.longitude * (</span><span style="color:#d08770;">180.0 </span><span>/ </span><span style="color:#d08770;">PI</span><span>);
</span><span> (latitude, longitude)
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">convert_to_uv_mercator</span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> (</span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#b48ead;">f32</span><span>) {
</span><span> </span><span style="color:#b48ead;">let </span><span>(lat, lon) = </span><span style="color:#bf616a;">self</span><span>.</span><span style="color:#96b5b4;">as_degrees</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> v = </span><span style="color:#96b5b4;">map_latitude</span><span>(lat).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> u = </span><span style="color:#96b5b4;">map_longitude</span><span>(lon).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> (u, v)
</span><span> }
</span><span>
</span><span> #[</span><span style="color:#bf616a;">allow</span><span>(dead_code)]
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">from_degrees</span><span>(</span><span style="color:#bf616a;">latitude</span><span>: </span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#bf616a;">longitude</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> Result<</span><span style="color:#b48ead;">Self</span><span>, CoordError> {
</span><span> </span><span style="color:#b48ead;">if </span><span>!(-</span><span style="color:#d08770;">90.0</span><span>..=</span><span style="color:#d08770;">90.0</span><span>).</span><span style="color:#96b5b4;">contains</span><span>(&latitude) {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(CoordError {
</span><span> msg: "</span><span style="color:#a3be8c;">Invalid latitude: {lat:?}</span><span>".</span><span style="color:#96b5b4;">to_string</span><span>(),
</span><span> });
</span><span> }
</span><span> </span><span style="color:#b48ead;">if </span><span>!(-</span><span style="color:#d08770;">180.0</span><span>..=</span><span style="color:#d08770;">180.0</span><span>).</span><span style="color:#96b5b4;">contains</span><span>(&longitude) {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(CoordError {
</span><span> msg: "</span><span style="color:#a3be8c;">Invalid longitude: {lon:?}</span><span>".</span><span style="color:#96b5b4;">to_string</span><span>(),
</span><span> });
</span><span> }
</span><span> </span><span style="color:#b48ead;">let</span><span> latitude = latitude / (</span><span style="color:#d08770;">180.0 </span><span>/ </span><span style="color:#d08770;">PI</span><span>);
</span><span> </span><span style="color:#b48ead;">let</span><span> longitude = longitude / (</span><span style="color:#d08770;">180.0 </span><span>/ </span><span style="color:#d08770;">PI</span><span>);
</span><span> Ok(Coordinates {
</span><span> latitude,
</span><span> longitude,
</span><span> })
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">get_point_on_sphere</span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> Vec3 {
</span><span> </span><span style="color:#b48ead;">let</span><span> y = </span><span style="color:#bf616a;">self</span><span>.latitude.</span><span style="color:#96b5b4;">sin</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> r = </span><span style="color:#bf616a;">self</span><span>.latitude.</span><span style="color:#96b5b4;">cos</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> x = </span><span style="color:#bf616a;">self</span><span>.longitude.</span><span style="color:#96b5b4;">sin</span><span>() * r;
</span><span> </span><span style="color:#b48ead;">let</span><span> z = </span><span style="color:#bf616a;">self</span><span>.longitude.</span><span style="color:#96b5b4;">cos</span><span>() * r;
</span><span> Vec3::new(x, y, z).</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS
</span><span> }
</span><span>}
</span></code></pre>
<p>So if we wanted to visualize cities with a population over 1 million as spheres, colored and sized based on total population, we could implement a system like this:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">spawn_city_population_spheres</span><span>(
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">commands</span><span>: Commands,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">meshes</span><span>: ResMut<Assets<Mesh>>,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">materials</span><span>: ResMut<Assets<StandardMaterial>>,
</span><span>) {
</span><span> </span><span style="color:#65737e;">// Cities data: (name, latitude, longitude, population in millions)
</span><span> </span><span style="color:#b48ead;">let</span><span> major_cities: Vec<(String, </span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#b48ead;">f32</span><span>)> = vec![
</span><span> (String::from("</span><span style="color:#a3be8c;">Tokyo</span><span>"), </span><span style="color:#d08770;">35.6762</span><span>, </span><span style="color:#d08770;">139.6503</span><span>, </span><span style="color:#d08770;">37.4</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Delhi</span><span>"), </span><span style="color:#d08770;">28.6139</span><span>, </span><span style="color:#d08770;">77.2090</span><span>, </span><span style="color:#d08770;">32.9</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Shanghai</span><span>"), </span><span style="color:#d08770;">31.2304</span><span>, </span><span style="color:#d08770;">121.4737</span><span>, </span><span style="color:#d08770;">28.5</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">São Paulo</span><span>"), -</span><span style="color:#d08770;">23.5505</span><span>, -</span><span style="color:#d08770;">46.6333</span><span>, </span><span style="color:#d08770;">22.4</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Mexico City</span><span>"), </span><span style="color:#d08770;">19.4326</span><span>, -</span><span style="color:#d08770;">99.1332</span><span>, </span><span style="color:#d08770;">22.2</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Cairo</span><span>"), </span><span style="color:#d08770;">30.0444</span><span>, </span><span style="color:#d08770;">31.2357</span><span>, </span><span style="color:#d08770;">21.3</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Mumbai</span><span>"), </span><span style="color:#d08770;">19.0760</span><span>, </span><span style="color:#d08770;">72.8777</span><span>, </span><span style="color:#d08770;">20.7</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Beijing</span><span>"), </span><span style="color:#d08770;">39.9042</span><span>, </span><span style="color:#d08770;">116.4074</span><span>, </span><span style="color:#d08770;">20.5</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Dhaka</span><span>"), </span><span style="color:#d08770;">23.8103</span><span>, </span><span style="color:#d08770;">90.4125</span><span>, </span><span style="color:#d08770;">19.6</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Osaka</span><span>"), </span><span style="color:#d08770;">34.6937</span><span>, </span><span style="color:#d08770;">135.5023</span><span>, </span><span style="color:#d08770;">19.2</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">New York</span><span>"), </span><span style="color:#d08770;">40.7128</span><span>, -</span><span style="color:#d08770;">74.0060</span><span>, </span><span style="color:#d08770;">18.8</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Karachi</span><span>"), </span><span style="color:#d08770;">24.8607</span><span>, </span><span style="color:#d08770;">67.0011</span><span>, </span><span style="color:#d08770;">16.5</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Buenos Aires</span><span>"), -</span><span style="color:#d08770;">34.6037</span><span>, -</span><span style="color:#d08770;">58.3816</span><span>, </span><span style="color:#d08770;">15.2</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Istanbul</span><span>"), </span><span style="color:#d08770;">41.0082</span><span>, </span><span style="color:#d08770;">28.9784</span><span>, </span><span style="color:#d08770;">15.1</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Kolkata</span><span>"), </span><span style="color:#d08770;">22.5726</span><span>, </span><span style="color:#d08770;">88.3639</span><span>, </span><span style="color:#d08770;">14.9</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Lagos</span><span>"), </span><span style="color:#d08770;">6.5244</span><span>, </span><span style="color:#d08770;">3.3792</span><span>, </span><span style="color:#d08770;">14.8</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">London</span><span>"), </span><span style="color:#d08770;">51.5074</span><span>, -</span><span style="color:#d08770;">0.1278</span><span>, </span><span style="color:#d08770;">14.3</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Los Angeles</span><span>"), </span><span style="color:#d08770;">34.0522</span><span>, -</span><span style="color:#d08770;">118.2437</span><span>, </span><span style="color:#d08770;">13.2</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Manila</span><span>"), </span><span style="color:#d08770;">14.5995</span><span>, </span><span style="color:#d08770;">120.9842</span><span>, </span><span style="color:#d08770;">13.1</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Rio de Janeiro</span><span>"), -</span><span style="color:#d08770;">22.9068</span><span>, -</span><span style="color:#d08770;">43.1729</span><span>, </span><span style="color:#d08770;">13.0</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Tianjin</span><span>"), </span><span style="color:#d08770;">39.3434</span><span>, </span><span style="color:#d08770;">117.3616</span><span>, </span><span style="color:#d08770;">12.8</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Kinshasa</span><span>"), -</span><span style="color:#d08770;">4.4419</span><span>, </span><span style="color:#d08770;">15.2663</span><span>, </span><span style="color:#d08770;">12.6</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Paris</span><span>"), </span><span style="color:#d08770;">48.8566</span><span>, </span><span style="color:#d08770;">2.3522</span><span>, </span><span style="color:#d08770;">11.1</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Shenzhen</span><span>"), </span><span style="color:#d08770;">22.5431</span><span>, </span><span style="color:#d08770;">114.0579</span><span>, </span><span style="color:#d08770;">10.6</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Jakarta</span><span>"), -</span><span style="color:#d08770;">6.2088</span><span>, </span><span style="color:#d08770;">106.8456</span><span>, </span><span style="color:#d08770;">10.6</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Bangalore</span><span>"), </span><span style="color:#d08770;">12.9716</span><span>, </span><span style="color:#d08770;">77.5946</span><span>, </span><span style="color:#d08770;">10.5</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Moscow</span><span>"), </span><span style="color:#d08770;">55.7558</span><span>, </span><span style="color:#d08770;">37.6173</span><span>, </span><span style="color:#d08770;">10.5</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Chennai</span><span>"), </span><span style="color:#d08770;">13.0827</span><span>, </span><span style="color:#d08770;">80.2707</span><span>, </span><span style="color:#d08770;">10.0</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Lima</span><span>"), -</span><span style="color:#d08770;">12.0464</span><span>, -</span><span style="color:#d08770;">77.0428</span><span>, </span><span style="color:#d08770;">9.7</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Bangkok</span><span>"), </span><span style="color:#d08770;">13.7563</span><span>, </span><span style="color:#d08770;">100.5018</span><span>, </span><span style="color:#d08770;">9.6</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Seoul</span><span>"), </span><span style="color:#d08770;">37.5665</span><span>, </span><span style="color:#d08770;">126.9780</span><span>, </span><span style="color:#d08770;">9.5</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Hyderabad</span><span>"), </span><span style="color:#d08770;">17.3850</span><span>, </span><span style="color:#d08770;">78.4867</span><span>, </span><span style="color:#d08770;">9.5</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Chengdu</span><span>"), </span><span style="color:#d08770;">30.5728</span><span>, </span><span style="color:#d08770;">104.0668</span><span>, </span><span style="color:#d08770;">9.3</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Singapore</span><span>"), </span><span style="color:#d08770;">1.3521</span><span>, </span><span style="color:#d08770;">103.8198</span><span>, </span><span style="color:#d08770;">5.7</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Ho Chi Minh City</span><span>"), </span><span style="color:#d08770;">10.8231</span><span>, </span><span style="color:#d08770;">106.6297</span><span>, </span><span style="color:#d08770;">9.1</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Toronto</span><span>"), </span><span style="color:#d08770;">43.6532</span><span>, -</span><span style="color:#d08770;">79.3832</span><span>, </span><span style="color:#d08770;">6.4</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Sydney</span><span>"), -</span><span style="color:#d08770;">33.8688</span><span>, </span><span style="color:#d08770;">151.2093</span><span>, </span><span style="color:#d08770;">5.3</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Johannesburg</span><span>"), -</span><span style="color:#d08770;">26.2041</span><span>, </span><span style="color:#d08770;">28.0473</span><span>, </span><span style="color:#d08770;">5.9</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Chicago</span><span>"), </span><span style="color:#d08770;">41.8781</span><span>, -</span><span style="color:#d08770;">87.6298</span><span>, </span><span style="color:#d08770;">8.9</span><span>),
</span><span> (String::from("</span><span style="color:#a3be8c;">Taipei</span><span>"), </span><span style="color:#d08770;">25.0330</span><span>, </span><span style="color:#d08770;">121.5654</span><span>, </span><span style="color:#d08770;">7.4</span><span>),
</span><span> ];
</span><span>
</span><span> </span><span style="color:#65737e;">// Define constants for scaling the spheres
</span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">BASE_RADIUS</span><span>: </span><span style="color:#b48ead;">f32 </span><span>= </span><span style="color:#d08770;">2.0</span><span>; </span><span style="color:#65737e;">// Minimum radius for smallest city
</span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">SCALE_FACTOR</span><span>: </span><span style="color:#b48ead;">f32 </span><span>= </span><span style="color:#d08770;">0.5</span><span>; </span><span style="color:#65737e;">// Multiplier for population to radius conversion
</span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">MIN_POPULATION</span><span>: </span><span style="color:#b48ead;">f32 </span><span>= </span><span style="color:#d08770;">5.0</span><span>; </span><span style="color:#65737e;">// For normalization purposes
</span><span> </span><span style="color:#b48ead;">const </span><span style="color:#d08770;">MAX_POPULATION</span><span>: </span><span style="color:#b48ead;">f32 </span><span>= </span><span style="color:#d08770;">40.0</span><span>; </span><span style="color:#65737e;">// For normalization purposes
</span><span>
</span><span> </span><span style="color:#65737e;">// Create a component to store city information.
</span><span> </span><span style="color:#65737e;">// Not used in this example, but could be used for a tooltip or similar.
</span><span> #[</span><span style="color:#bf616a;">derive</span><span>(Component)]
</span><span> </span><span style="color:#b48ead;">struct </span><span>CityMarker {
</span><span> </span><span style="color:#bf616a;">name</span><span>: String,
</span><span> </span><span style="color:#bf616a;">population</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">// Create a mesh that will be reused for all cities
</span><span> </span><span style="color:#b48ead;">let</span><span> sphere_mesh = meshes.</span><span style="color:#96b5b4;">add</span><span>(
</span><span> Mesh::try_from(shape::Icosphere {
</span><span> radius: </span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#65737e;">// We'll scale this in the transform
</span><span> subdivisions: </span><span style="color:#d08770;">32</span><span>,
</span><span> })
</span><span> .</span><span style="color:#96b5b4;">unwrap</span><span>(),
</span><span> );
</span><span>
</span><span> </span><span style="color:#65737e;">// Spawn a sphere for each city
</span><span> </span><span style="color:#b48ead;">for </span><span>(name, latitude, longitude, population) in major_cities {
</span><span> </span><span style="color:#65737e;">// Convert latitude and longitude to 3D coordinates on the sphere
</span><span> </span><span style="color:#b48ead;">let</span><span> coords = Coordinates::from_degrees(latitude, longitude)
</span><span> .</span><span style="color:#96b5b4;">unwrap</span><span>()
</span><span> .</span><span style="color:#96b5b4;">get_point_on_sphere</span><span>();
</span><span>
</span><span> </span><span style="color:#65737e;">// Calculate sphere size based on population
</span><span> </span><span style="color:#65737e;">// Using a logarithmic scale to prevent extremely large cities from dominating
</span><span> </span><span style="color:#b48ead;">let</span><span> normalized_population =
</span><span> (population - </span><span style="color:#d08770;">MIN_POPULATION</span><span>) / (</span><span style="color:#d08770;">MAX_POPULATION </span><span>- </span><span style="color:#d08770;">MIN_POPULATION</span><span>);
</span><span> </span><span style="color:#b48ead;">let</span><span> size = </span><span style="color:#d08770;">BASE_RADIUS </span><span>+ (normalized_population * </span><span style="color:#d08770;">SCALE_FACTOR </span><span>* </span><span style="color:#d08770;">10.0</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Calculate color based on population (gradient from yellow to red)
</span><span> </span><span style="color:#b48ead;">let</span><span> t = normalized_population.</span><span style="color:#96b5b4;">clamp</span><span>(</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">1.0</span><span>);
</span><span> </span><span style="color:#b48ead;">let</span><span> color = Color::rgb(
</span><span> </span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#65737e;">// Red stays at 1.0
</span><span> </span><span style="color:#d08770;">1.0 </span><span>- (t * </span><span style="color:#d08770;">0.7</span><span>), </span><span style="color:#65737e;">// Green decreases with population
</span><span> </span><span style="color:#d08770;">0.5 </span><span>- (t * </span><span style="color:#d08770;">0.4</span><span>), </span><span style="color:#65737e;">// Blue decreases with population
</span><span> );
</span><span>
</span><span> </span><span style="color:#65737e;">// Spawn the city sphere
</span><span> commands.</span><span style="color:#96b5b4;">spawn</span><span>((
</span><span> PbrBundle {
</span><span> mesh: sphere_mesh.</span><span style="color:#96b5b4;">clone</span><span>(),
</span><span> material: materials.</span><span style="color:#96b5b4;">add</span><span>(StandardMaterial {
</span><span> base_color: color,
</span><span> unlit: </span><span style="color:#d08770;">true</span><span>,
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> }),
</span><span> transform: Transform::from_translation(Vec3::new(coords.x, coords.y, coords.z))
</span><span> .</span><span style="color:#96b5b4;">with_scale</span><span>(Vec3::splat(size)),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> },
</span><span> CityMarker { name, population },
</span><span> ));
</span><span> }
</span><span>}
</span></code></pre>
<p>Which will render a lovely scaled and colored sphere on each of our coordinates.</p>
<img src="../../images/bevy_world_pop_map.png" alt="Map visualizing population in major cities" />
<p>In closing, I hope this is informative. I'd like to eventually make this a bit more general and re-usable. Perhaps releasing it as a library or as a WASM app in the future. Building this was an incredible learning experience though. I was a complete newbie to GIS concepts before this. And knowing how to read from Geographic raster files is an incredibly useful thing that can be utilized in lots of different software engineering domains.</p>
Bevy Procedural Earth Part 2: Coordinate Systems and Materials2023-10-06T00:00:00+00:002023-10-06T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/bevy-proc-earth-2/<img src="../../images/earth_banner_2.png" alt="A boring, untextured, sphere." />
<p>In part <a href="https://blog.graysonhead.net/posts/bevy-proc-earth-1/">one</a> of this project, we went over the basics of mathematically constructing a mesh. At the end of it we were left with a segmented, but very boring grey sphere:</p>
<img src="../../images/plain_mesh_sphere.png" alt="A boring, untextured, sphere." />
<p>Obviously this is no fun, we want to get some textures on it. </p>
<p>Now, there are lots of ways we could go about wrapping our texture, but I'm going to go about this in a somewhat unusual way. Since we are texturing a planet, and we have a lot of different maps with different projections to pick from, we are going to just select an appropriate one and calculate all our texture coordinates with latitude/longitude coordinates.</p>
<h1 id="coordinate-systems-disambiguated">Coordinate Systems Disambiguated</h1>
<p>By the end of this article, we will have 3 different coordinate systems involved in procedurally generating our facsimile earth, they are:</p>
<h3 id="bevy-s-cartesian-coordinates">Bevy's Cartesian Coordinates</h3>
<p>These are the X, Y, Z triplet that identifies the location of the Entity in 3d space. There is <a href="https://docs.rs/bevy/latest/bevy/transform/components/struct.GlobalTransform.html">GlobalTransform</a>, which is the absolute position of the Entity relative to the origin <code>(0, 0, 0)</code>. And the <a href="https://docs.rs/bevy/latest/bevy/transform/components/struct.Transform.html">Transform</a>, which is the position of the Entity relative to its parent (or the origin, if it has no parent.)</p>
<p>This is the coordinate space that our individual vertices will exist in.</p>
<h3 id="uv-texture-coordinates">UV (Texture) Coordinates</h3>
<p>UV coordinates how flat textures are mapped onto the 3d surface of a mesh.</p>
<img src="https://upload.wikimedia.org/wikipedia/commons/0/04/UVMapping.png" alt="UV Mapping Graph from Wikipedia https://creativecommons.org/licenses/by-sa/3.0/">
<p>What U and V actually are, are a 2d coordinate system (usually from 0.0 to 1.0, but can also be by raw pixel count in some engines/apis.) We will assign each of our vertexes in 3d space onto this 2d coordinate system, and then the relevant segment of the 2d texture will be projected on each triangle that we generated in the last tutorial. </p>
<h3 id="geodetic-coordinates">Geodetic Coordinates</h3>
<p>Geodetic Coordinates are a pair of angles that describe the location of an object on the surface of a sphere. </p>
<img src="../../images/Lat_Lon.svg" alt="" />
<p>In reality, they aren't this simple. The objects we normally measure with Geodetic coordinates (such as earth) are not typically perfectly spherical. As a result of this (and computers), we have the wonderful field of GIS. (This has made a lot of software engineers very confused, and has been widely regarded as a bad move.) The variety of actual Geodetic systems, and their insanely large number of Datums (reference points from which spatial measurements are made) can make your head spin.</p>
<p>However, you can go far in the field of GIS by pretending that earth is a sphere (<a href="https://www.sciencedirect.com/topics/earth-and-planetary-sciences/oblate-spheroids">which it definitely isn't</a>) and pretending all datums are WGS84 (or better yet, convert them). <sup class="footnote-reference"><a href="#1">1</a></sup> </p>
<p>But for this project, it doesn't matter. We aren't optimizing for 10m accuracy, we just want a pretty earth.</p>
<h2 id="converting-from-cartesian-to-uv">Converting from Cartesian to UV</h2>
<p>Fortunately we have a somewhat direct method of converting between our Cartesian Coordinates and UV coordinates. In this example, we are going to use Geodetic Coordinates as a translation between them. </p>
<p>Knowing that Positive longitudes are east of the Prime Meridian, negative west of the Prime Meridian, and Positive and Negative latitudes are north and south of the equator respectively, we can flatten these onto a 2d map with values that look like this:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> -180 (W) 0 180 (E)
</span><span> ┌────────────────────────────────────────────────┐
</span><span> +90 (N)│ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> 0 │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> -90 (S)└────────────────────────────────────────────────┘
</span></code></pre>
<p>Which is more or less how Mercator projection works.</p>
<p>All we need to do to convert a latitude/longitude pair to a position on our UV map. To do so, need to map the latitude and longitude from their normal ranges to 0.0 -> 1.0. Which is accomplished by this pair of functions:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">map_latitude</span><span>(</span><span style="color:#bf616a;">lat</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> Result<</span><span style="color:#b48ead;">f32</span><span>, CoordError> {
</span><span> </span><span style="color:#65737e;">// 90 -> 0 maps to 0.0 to 0.5
</span><span> </span><span style="color:#65737e;">// 0 -> -90 maps to 0.5 to 1.0
</span><span> </span><span style="color:#65737e;">// Ensure latitude is valid
</span><span> </span><span style="color:#b48ead;">if </span><span>!(-</span><span style="color:#d08770;">90.0</span><span>..=</span><span style="color:#d08770;">90.0</span><span>).</span><span style="color:#96b5b4;">contains</span><span>(&lat) {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(CoordError {
</span><span> msg: "</span><span style="color:#a3be8c;">Invalid latitude: {lat:?}</span><span>".</span><span style="color:#96b5b4;">to_string</span><span>(),
</span><span> });
</span><span> }
</span><span> </span><span style="color:#b48ead;">if </span><span>(</span><span style="color:#d08770;">90.0</span><span>..=</span><span style="color:#d08770;">0.0</span><span>).</span><span style="color:#96b5b4;">contains</span><span>(&lat) {
</span><span> Ok(</span><span style="color:#96b5b4;">map</span><span>((</span><span style="color:#d08770;">90.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>), (</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">0.5</span><span>), lat))
</span><span> } </span><span style="color:#b48ead;">else </span><span>{
</span><span> Ok(</span><span style="color:#96b5b4;">map</span><span>((</span><span style="color:#d08770;">0.0</span><span>, -</span><span style="color:#d08770;">90.0</span><span>), (</span><span style="color:#d08770;">0.5</span><span>, </span><span style="color:#d08770;">1.0</span><span>), lat))
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">map_longitude</span><span>(</span><span style="color:#bf616a;">lon</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> Result<</span><span style="color:#b48ead;">f32</span><span>, CoordError> {
</span><span> </span><span style="color:#65737e;">// -180 -> 0 maps to 0.0 to 0.5
</span><span> </span><span style="color:#65737e;">// 0 -> 180 maps to 0.5 to 1.0
</span><span> </span><span style="color:#65737e;">//Ensure longitude is valid
</span><span> </span><span style="color:#b48ead;">if </span><span>!(-</span><span style="color:#d08770;">180.0</span><span>..=</span><span style="color:#d08770;">180.0</span><span>).</span><span style="color:#96b5b4;">contains</span><span>(&lon) {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(CoordError {
</span><span> msg: "</span><span style="color:#a3be8c;">Invalid longitude: {lon:?}</span><span>".</span><span style="color:#96b5b4;">to_string</span><span>(),
</span><span> });
</span><span> }
</span><span> </span><span style="color:#b48ead;">if </span><span>(-</span><span style="color:#d08770;">180.0</span><span>..=</span><span style="color:#d08770;">0.0</span><span>).</span><span style="color:#96b5b4;">contains</span><span>(&lon) {
</span><span> Ok(</span><span style="color:#96b5b4;">map</span><span>((-</span><span style="color:#d08770;">180.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>), (</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">0.5</span><span>), lon))
</span><span> } </span><span style="color:#b48ead;">else </span><span>{
</span><span> Ok(</span><span style="color:#96b5b4;">map</span><span>((</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">180.0</span><span>), (</span><span style="color:#d08770;">0.5</span><span>, </span><span style="color:#d08770;">1.0</span><span>), lon))
</span><span> }
</span><span>}
</span></code></pre>
<p>Now, the last piece we need to make this coordinate conversion work is the ability to convert Cartesian Coordinates (x, y, z) to Geodetic ones (lat, lon). Assuming the point is normalized, getting these two angles is not difficult.</p>
<p>For latitude, we need to take the arcsine of the point of our "up" axis, which in bevy is positive Y. For longitude, we need the arctangent of our two "horizontal" axes, X and Z. In rust, that looks like:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> latitude = normalized_point.y.</span><span style="color:#96b5b4;">asin</span><span>();
</span><span style="color:#b48ead;">let</span><span> longitude = normalized_point.x.</span><span style="color:#96b5b4;">atan2</span><span>(normalized_point.z);
</span></code></pre>
<p>This quite easily gets our latitude and longitude in radians. Note that this strategy only works if the GlobalTransform of the sphere is precisely at the origin <code>(0, 0, 0)</code> of our scene. If you want this
to work on spheres not at the origin of the scene (or for multiple different spheres) there will be a few extra steps involved.</p>
<p>I also want some easy ways to convert to degrees, since latitude and longitude aren't typically represented as radians. Fortunately this conversion is simple and not terribly lossy, so lets package all of this (as well as a few helper methods) up into a struct:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#bf616a;">derive</span><span>(Debug)]
</span><span style="color:#b48ead;">pub struct </span><span>Coordinates {
</span><span> </span><span style="color:#65737e;">// Stored internally in radians
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">latitude</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#b48ead;">pub </span><span style="color:#bf616a;">longitude</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>From<Vec3> </span><span style="color:#b48ead;">for </span><span>Coordinates {
</span><span> </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">from</span><span>(</span><span style="color:#bf616a;">value</span><span>: Vec3) -> </span><span style="color:#b48ead;">Self </span><span>{
</span><span> </span><span style="color:#b48ead;">let</span><span> normalized_point = value.</span><span style="color:#96b5b4;">normalize</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> latitude = normalized_point.y.</span><span style="color:#96b5b4;">asin</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> longitude = normalized_point.x.</span><span style="color:#96b5b4;">atan2</span><span>(normalized_point.z);
</span><span> Coordinates {
</span><span> latitude,
</span><span> longitude,
</span><span> }
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">impl </span><span>Coordinates {
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">as_degrees</span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> (</span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#b48ead;">f32</span><span>) {
</span><span> </span><span style="color:#b48ead;">let</span><span> latitude = </span><span style="color:#bf616a;">self</span><span>.latitude * (</span><span style="color:#d08770;">180.0 </span><span>/ </span><span style="color:#d08770;">PI</span><span>);
</span><span> </span><span style="color:#b48ead;">let</span><span> longitude = </span><span style="color:#bf616a;">self</span><span>.longitude * (</span><span style="color:#d08770;">180.0 </span><span>/ </span><span style="color:#d08770;">PI</span><span>);
</span><span> (latitude, longitude)
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">convert_to_uv_mercator</span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> (</span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#b48ead;">f32</span><span>) {
</span><span> </span><span style="color:#b48ead;">let </span><span>(lat, lon) = </span><span style="color:#bf616a;">self</span><span>.</span><span style="color:#96b5b4;">as_degrees</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> v = </span><span style="color:#96b5b4;">map_latitude</span><span>(lat).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> u = </span><span style="color:#96b5b4;">map_longitude</span><span>(lon).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> (u, v)
</span><span> }
</span><span>
</span><span> #[</span><span style="color:#bf616a;">allow</span><span>(dead_code)]
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">from_degrees</span><span>(</span><span style="color:#bf616a;">latitude</span><span>: </span><span style="color:#b48ead;">f32</span><span>, </span><span style="color:#bf616a;">longitude</span><span>: </span><span style="color:#b48ead;">f32</span><span>) -> Result<</span><span style="color:#b48ead;">Self</span><span>, CoordError> {
</span><span> </span><span style="color:#b48ead;">if </span><span>!(-</span><span style="color:#d08770;">90.0</span><span>..=</span><span style="color:#d08770;">90.0</span><span>).</span><span style="color:#96b5b4;">contains</span><span>(&latitude) {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(CoordError {
</span><span> msg: "</span><span style="color:#a3be8c;">Invalid latitude: {lat:?}</span><span>".</span><span style="color:#96b5b4;">to_string</span><span>(),
</span><span> });
</span><span> }
</span><span> </span><span style="color:#b48ead;">if </span><span>!(-</span><span style="color:#d08770;">180.0</span><span>..=</span><span style="color:#d08770;">180.0</span><span>).</span><span style="color:#96b5b4;">contains</span><span>(&longitude) {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(CoordError {
</span><span> msg: "</span><span style="color:#a3be8c;">Invalid longitude: {lon:?}</span><span>".</span><span style="color:#96b5b4;">to_string</span><span>(),
</span><span> });
</span><span> }
</span><span> </span><span style="color:#b48ead;">let</span><span> latitude = latitude / (</span><span style="color:#d08770;">180.0 </span><span>/ </span><span style="color:#d08770;">PI</span><span>);
</span><span> </span><span style="color:#b48ead;">let</span><span> longitude = longitude / (</span><span style="color:#d08770;">180.0 </span><span>/ </span><span style="color:#d08770;">PI</span><span>);
</span><span> Ok(Coordinates {
</span><span> latitude,
</span><span> longitude,
</span><span> })
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">get_point_on_sphere</span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> Vec3 {
</span><span> </span><span style="color:#b48ead;">let</span><span> y = </span><span style="color:#bf616a;">self</span><span>.latitude.</span><span style="color:#96b5b4;">sin</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> r = </span><span style="color:#bf616a;">self</span><span>.latitude.</span><span style="color:#96b5b4;">cos</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> x = </span><span style="color:#bf616a;">self</span><span>.longitude.</span><span style="color:#96b5b4;">sin</span><span>() * -r;
</span><span> </span><span style="color:#b48ead;">let</span><span> z = </span><span style="color:#bf616a;">self</span><span>.longitude.</span><span style="color:#96b5b4;">cos</span><span>() * r;
</span><span> Vec3::new(x, y, z).</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS
</span><span> }
</span><span>}
</span></code></pre>
<p>You might also notice that the <code>convert_to_uv_mercator</code> method calls the mapping functions we made earlier, allowing us to convert between latlon and UV just by calling this method on a point.</p>
<p>In addition, we have a method <code>get_point_on_sphere</code>:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">get_point_on_sphere</span><span>(&</span><span style="color:#bf616a;">self</span><span>) -> Vec3 {
</span><span> </span><span style="color:#b48ead;">let</span><span> y = </span><span style="color:#bf616a;">self</span><span>.latitude.</span><span style="color:#96b5b4;">sin</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> r = </span><span style="color:#bf616a;">self</span><span>.latitude.</span><span style="color:#96b5b4;">cos</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> x = </span><span style="color:#bf616a;">self</span><span>.longitude.</span><span style="color:#96b5b4;">sin</span><span>() * -r;
</span><span> </span><span style="color:#b48ead;">let</span><span> z = </span><span style="color:#bf616a;">self</span><span>.longitude.</span><span style="color:#96b5b4;">cos</span><span>() * r;
</span><span> Vec3::new(x, y, z).</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS
</span><span>}
</span></code></pre>
<p>This allows us to convert in the other direction, and get a point on the sphere from Geodetic coordinates. This would be useful if you want to spawn an object at a given latlon.</p>
<h2 id="setting-uv-coordinates">Setting UV Coordinates</h2>
<p>Now that we have a means to get a UV coordinate for each vertex in our mesh, we need to actually assign them. So lets take a look at a new version of our <code>generate_face</code> function:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">generate_face</span><span>(
</span><span> </span><span style="color:#bf616a;">normal</span><span>: Vec3,
</span><span> </span><span style="color:#bf616a;">resolution</span><span>: </span><span style="color:#b48ead;">u32</span><span>,
</span><span> </span><span style="color:#bf616a;">x_offset</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#bf616a;">y_offset</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span>) -> Mesh {
</span><span> </span><span style="color:#b48ead;">let</span><span> axis_a = Vec3::new(normal.y, normal.z, normal.x); </span><span style="color:#65737e;">// Horizontal
</span><span> </span><span style="color:#b48ead;">let</span><span> axis_b = axis_a.</span><span style="color:#96b5b4;">cross</span><span>(normal); </span><span style="color:#65737e;">// Vertical
</span><span> </span><span style="color:#b48ead;">let mut</span><span> verticies: Vec<Vec3> = Vec::new();
</span><span>
</span><span style="color:#65737e;">// Create a new vec containing our uv coords
</span><span> </span><span style="color:#b48ead;">let mut</span><span> uvs = Vec::new();
</span><span>
</span><span> </span><span style="color:#b48ead;">let mut</span><span> indicies: Vec<</span><span style="color:#b48ead;">u32</span><span>> = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> normals = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> first_longitude = </span><span style="color:#d08770;">0.0</span><span>;
</span><span> </span><span style="color:#b48ead;">for</span><span> y in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">for</span><span> x in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">let</span><span> i = x + y * resolution;
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> percent = Vec2::new(x as </span><span style="color:#b48ead;">f32</span><span>, y as </span><span style="color:#b48ead;">f32</span><span>) / (resolution - </span><span style="color:#d08770;">1</span><span>) as </span><span style="color:#b48ead;">f32</span><span>;
</span><span> </span><span style="color:#b48ead;">let</span><span> point_on_unit_cube =
</span><span> normal + (percent.x - x_offset) * axis_a + (percent.y - y_offset) * axis_b;
</span><span> </span><span style="color:#65737e;">// Convert our point_coords into `Coordinates`
</span><span> </span><span style="color:#b48ead;">let</span><span> point_coords: Coordinates = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>().</span><span style="color:#96b5b4;">into</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> normalized_point = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS</span><span>;
</span><span> verticies.</span><span style="color:#96b5b4;">push</span><span>(normalized_point);
</span><span> </span><span style="color:#65737e;">// Get the UV Coordinates of our point
</span><span> </span><span style="color:#b48ead;">let </span><span>(</span><span style="color:#b48ead;">mut</span><span> u, </span><span style="color:#b48ead;">mut</span><span> v) = point_coords.</span><span style="color:#96b5b4;">convert_to_uv_mercator</span><span>();
</span><span>
</span><span> </span><span style="color:#65737e;">// Some special case logic
</span><span> </span><span style="color:#b48ead;">if</span><span> y == </span><span style="color:#d08770;">0 </span><span>&& x == </span><span style="color:#d08770;">0 </span><span>{
</span><span> first_longitude = lon;
</span><span> }
</span><span> </span><span style="color:#65737e;">// In the middle latitudes, if we start on a
</span><span> </span><span style="color:#65737e;">// negative longitude but then wind up crossing to a
</span><span> </span><span style="color:#65737e;">// positive longitude, set u to 0.0 to prevent a seam
</span><span> </span><span style="color:#b48ead;">if</span><span> first_longitude < </span><span style="color:#d08770;">0.0 </span><span>&& lon > </span><span style="color:#d08770;">0.0 </span><span>&& lat < </span><span style="color:#d08770;">89.0 </span><span>&& lat > -</span><span style="color:#d08770;">89.0 </span><span>{
</span><span> u = </span><span style="color:#d08770;">0.0</span><span>;
</span><span> }
</span><span> </span><span style="color:#65737e;">// If we are below -40 degrees latitude and the tile
</span><span> </span><span style="color:#65737e;">// starts at 180 degrees, set u to 0.0 to prevent a seam
</span><span> </span><span style="color:#b48ead;">if</span><span> x == </span><span style="color:#d08770;">0 </span><span>&& lon == </span><span style="color:#d08770;">180.0 </span><span>&& lat < -</span><span style="color:#d08770;">40.0 </span><span>{
</span><span> u = </span><span style="color:#d08770;">0.0</span><span>;
</span><span> }
</span><span> </span><span style="color:#65737e;">// Push the UV Coordinates into the vec
</span><span> uvs.</span><span style="color:#96b5b4;">push</span><span>([u, v]);
</span><span> normals.</span><span style="color:#96b5b4;">push</span><span>(-point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>());
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> x != resolution - </span><span style="color:#d08770;">1 </span><span>&& y != resolution - </span><span style="color:#d08770;">1 </span><span>{
</span><span> </span><span style="color:#65737e;">// First triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Second triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + </span><span style="color:#d08770;">1</span><span>);
</span><span> }
</span><span> }
</span><span> }
</span><span> </span><span style="color:#b48ead;">let</span><span> indicies = mesh::Indices::</span><span style="color:#d08770;">U32</span><span>(indicies);
</span><span> </span><span style="color:#b48ead;">let mut</span><span> mesh = Mesh::new(PrimitiveTopology::TriangleList);
</span><span> mesh.</span><span style="color:#96b5b4;">set_indices</span><span>(Some(indicies));
</span><span> mesh.</span><span style="color:#96b5b4;">insert_attribute</span><span>(Mesh::</span><span style="color:#d08770;">ATTRIBUTE_POSITION</span><span>, verticies);
</span><span> mesh.</span><span style="color:#96b5b4;">insert_attribute</span><span>(Mesh::</span><span style="color:#d08770;">ATTRIBUTE_NORMAL</span><span>, normals);
</span><span> </span><span style="color:#65737e;">// Insert the UV attribute along with our uv vec
</span><span> mesh.</span><span style="color:#96b5b4;">insert_attribute</span><span>(Mesh::</span><span style="color:#d08770;">ATTRIBUTE_UV_0</span><span>, uvs);
</span><span> mesh.</span><span style="color:#96b5b4;">generate_tangents</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> mesh
</span><span>}
</span></code></pre>
<p>You are probably wondering what is up with this bit specifically:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span> </span><span style="color:#65737e;">// Some special case logic
</span><span> </span><span style="color:#b48ead;">if</span><span> y == </span><span style="color:#d08770;">0 </span><span>&& x == </span><span style="color:#d08770;">0 </span><span>{
</span><span> first_longitude = lon;
</span><span> }
</span><span> </span><span style="color:#65737e;">// In the middle latitudes, if we start on a
</span><span> </span><span style="color:#65737e;">// negative longitude but then wind up crossing to a
</span><span> </span><span style="color:#65737e;">// positive longitude, set u to 0.0 to prevent a seam
</span><span> </span><span style="color:#b48ead;">if</span><span> first_longitude < </span><span style="color:#d08770;">0.0 </span><span>&& lon > </span><span style="color:#d08770;">0.0 </span><span>&& lat < </span><span style="color:#d08770;">89.0 </span><span>&& lat > -</span><span style="color:#d08770;">89.0 </span><span>{
</span><span> u = </span><span style="color:#d08770;">0.0</span><span>;
</span><span> }
</span><span> </span><span style="color:#65737e;">// If we are below -40 degrees latitude and the tile
</span><span> </span><span style="color:#65737e;">// starts at 180 degrees, set u to 0.0 to prevent a seam
</span><span> </span><span style="color:#b48ead;">if</span><span> x == </span><span style="color:#d08770;">0 </span><span>&& lon == </span><span style="color:#d08770;">180.0 </span><span>&& lat < -</span><span style="color:#d08770;">40.0 </span><span>{
</span><span> u = </span><span style="color:#d08770;">0.0</span><span>;
</span><span> }
</span></code></pre>
<p>This is a section that is needed to prevent seams where the UV coordinates go in the wrong direction. Since the coordinates near the prime meridian go from
1 to 0 in the space of one vertex, the entire texture is re-projected backwards on the faces leading up to the prime meridian. There are a few ways to fix this, namely offsetting
your vertexes so that they are <em>exactly</em> aligned with the prime meridian, but I decided to just clamp the values close to both the prime meridian and the poles to prevent the seams from occurring.</p>
<p>And, just to test that everything is working properly, lets go ahead and assign it an <a href="https://www.google.com/search?client=firefox-b-1-d&q=uv+test+texture">uv test texture</a> of your choice.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">generate_faces</span><span>(
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">commands</span><span>: Commands,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">meshes</span><span>: ResMut<Assets<Mesh>>,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">materials</span><span>: ResMut<Assets<StandardMaterial>>,
</span><span> </span><span style="color:#bf616a;">asset_server</span><span>: Res<AssetServer>,
</span><span>) {
</span><span> </span><span style="color:#b48ead;">let</span><span> faces = vec![
</span><span> Vec3::X,
</span><span> Vec3::</span><span style="color:#d08770;">NEG_X</span><span>,
</span><span> Vec3::Y,
</span><span> Vec3::</span><span style="color:#d08770;">NEG_Y</span><span>,
</span><span> Vec3::Z,
</span><span> Vec3::</span><span style="color:#d08770;">NEG_Z</span><span>,
</span><span> ];
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> offsets = vec![(</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>), (</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">1.0</span><span>), (</span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>), (</span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#d08770;">1.0</span><span>)];
</span><span>
</span><span> </span><span style="color:#b48ead;">let mut</span><span> rng = rand::thread_rng();
</span><span>
</span><span> </span><span style="color:#b48ead;">for</span><span> direction in faces {
</span><span> </span><span style="color:#b48ead;">for</span><span> offset in &offsets {
</span><span> commands.</span><span style="color:#96b5b4;">spawn</span><span>((
</span><span> PbrBundle {
</span><span> mesh: meshes.</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#96b5b4;">generate_face</span><span>(direction, </span><span style="color:#d08770;">100</span><span>, offset.</span><span style="color:#d08770;">0</span><span>, offset.</span><span style="color:#d08770;">1</span><span>)),
</span><span> material: materials.</span><span style="color:#96b5b4;">add</span><span>(StandardMaterial {
</span><span> base_color_texture: Some(
</span><span> asset_server.</span><span style="color:#96b5b4;">load</span><span>("</span><span style="color:#a3be8c;">WorldTextures/uv_test_texture.png</span><span>"),
</span><span> ),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> }),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> },
</span><span> PickableBundle::default(), </span><span style="color:#65737e;">// Makes the entity pickable
</span><span> RaycastPickTarget::default(),
</span><span> On::<Pointer<Click>>::run(|</span><span style="color:#bf616a;">event</span><span>: Listener<Pointer<Click>>| {
</span><span> info!("</span><span style="color:#a3be8c;">Clicked on entity {:?}</span><span>", event);
</span><span> </span><span style="color:#b48ead;">let</span><span> hit = event.hit;
</span><span> </span><span style="color:#b48ead;">if let </span><span>Some(pos) = hit.position {
</span><span> </span><span style="color:#b48ead;">let</span><span> coords: Coordinates = pos.</span><span style="color:#96b5b4;">into</span><span>();
</span><span> </span><span style="color:#b48ead;">let </span><span>(latitude, longitude) = coords.</span><span style="color:#96b5b4;">as_degrees</span><span>();
</span><span> info!(
</span><span> "</span><span style="color:#a3be8c;">Latlon of selected point: Lat: {}, Lon: {}</span><span>",
</span><span> latitude, longitude
</span><span> );
</span><span> }
</span><span> }),
</span><span> ));
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<p>While we were messing with that function, I also used the wonderful <a href="https://github.com/aevyrie/bevy_mod_picking">bevy picking module</a> to give us a chance to look at/debug our coordinate generation if it isn't working correctly. Now when you click on the sphere, you should get a log message emitted with the latitude and longitude coordinates.</p>
<p>So, lets fire it up and see what happens:</p>
<video controls width=100%>
<source src="../../videos/sphere.webm" />
</video>
<p>Which looks pretty good!</p>
<p>And when you click on the globe, you get this lovely log message:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>2023-10-20T19:14:05.861205Z INFO surface_action::map: Latlon of selected point: Lat: 29.734213, Lon: -95.40474
</span></code></pre>
<p>So now that all of that is out of the way, lets make it look like an actual earth.</p>
<h2 id="textures">Textures</h2>
<p>Head over to <a href="https://visibleearth.nasa.gov/collection/1484/blue-marble?page=2">Visible Earth</a> and download the texture package for the month of your choice. Use the highest resolution one if possible for maximum pretty. But if you get a WGPU error when compiling, try a lower resolution (you may also have to resample them in GIMP or another program if you are running on a computer without a discrete GPU.)</p>
<p>It should look something like this:</p>
<img src="../../images/bevy_proc_earth_earth_color.jpg" alt="Earth color texture." />
<p>Next, we will add a metallic roughness texture. This may be a bit confusing, because we aren't creating a metallic object, but we are going to use this channel to differentiate between the reflectivity of land versus the ocean. As a result, I've taken one of NASAs specular maps and inverted it (you can download and invert it yourself from NASAs website, or just save the image below):</p>
<img src="../../images/specular_map_inverted_8k.png" alt="Specular map of the earth." />
<p>And finally for our normal channel, we have a 21k resolution topography map.</p>
<img src="../../images/topography_21K.png" alt="Topographicl map of the earth." />
<p>Since these images all use the same projection, the same UV coordinates will suffice for all of them. As a result, the only thing left to do is add them to our PBR material in the <code>generate_faces</code> function:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>PbrBundle {
</span><span> mesh: meshes.</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#96b5b4;">generate_face</span><span>(direction, </span><span style="color:#d08770;">800</span><span>, offset.</span><span style="color:#d08770;">0</span><span>, offset.</span><span style="color:#d08770;">1</span><span>, &rs)),
</span><span> material: materials.</span><span style="color:#96b5b4;">add</span><span>(StandardMaterial {
</span><span> base_color_texture: Some(
</span><span> asset_server.</span><span style="color:#96b5b4;">load</span><span>("</span><span style="color:#a3be8c;">WorldTextures/world_shaded_32k.png</span><span>"),
</span><span> ),
</span><span> metallic_roughness_texture: Some(
</span><span> asset_server.</span><span style="color:#96b5b4;">load</span><span>("</span><span style="color:#a3be8c;">WorldTextures/specular_map_inverted_8k.png</span><span>"),
</span><span> ),
</span><span> perceptual_roughness: </span><span style="color:#d08770;">1.0</span><span>,
</span><span> normal_map_texture: Some(
</span><span> asset_server.</span><span style="color:#96b5b4;">load</span><span>("</span><span style="color:#a3be8c;">WorldTextures/topography_21K.png</span><span>"),
</span><span> ),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> }),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span>},
</span></code></pre>
<p>And the results should be a textured Earth:</p>
<video controls width=100%>
<source src="../../videos/spinny_earth.webm" />
</video>
<p>Next time we are going to use some GIS real world elevation data to deform the mesh into an exaggerated height model of the earth.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Please don't actually do this if you are going into a GIS career though.</p>
</div>
Bevy Procedural Earth Part 1: Mesh2023-10-03T00:00:00+00:002023-10-03T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/bevy-proc-earth-1/<img src="../../images/earth_banner.png" alt="A sneak peek at the finished project; A picture of earth." />
<p>Of all the youtube game vloggers on the internet, <a href="https://www.youtube.com/@SebastianLague">Sebastian Lague</a> is probably my favorite. Even though I have never written anything in Unity (and probably never will, especially given recent events.) I still find a lot of his videos inspirational.</p>
<p>One of his <a href="https://youtu.be/sLqXFF8mlEU?si=YTEgdMKiDiSr2nzk">videos</a> in particular spurred me to do something I previously didn't have a whole lot of interest in: procedurally generating meshes. So I decided to go about implementing his idea in Bevy with a few twists.</p>
<p>While his video was a excellent starting spot, C# and Rust are drastically different languages so translating between them didn't always get me the results I expected. In addition, there were some bits he glossed over (forgivable, otherwise the video would have been 6 hours long) that I felt deserved a bit more explanation in text form.</p>
<p>Now, some disclaimers; I'm probably not going to be able to publish the full source of this project because the assets are far too large for Github. I may publish a torrent magnet link or something at some point, but if you want to follow along you will need to fill in some of the blank spots yourself as I'm going to be sharing piecemeal examples to illustrate some of the parts I found more difficult or less obvious (And yes, I know I just complained about people glossing over things. Sorry, not sorry.) If you have questions, feel free to reach out to me (<a href="https://hachyderm.io/@Darkside">mastadon</a> is probably the best way.)</p>
<p>Also, fair warning, the example code is not minimal in any way and will be quite dense. It took me weeks to learn this stuff, so unpacking it all in a blog post is bound to be an interesting exercise.</p>
<p>So, disclaimer out of the way, lets get started.</p>
<h1 id="from-planes-a-sphere">From Planes, A Sphere</h1>
<p>Like in Sebastian's video, I'm also going to construct my earth from a set of planes. However, I wanted to be able to subdivide the earth into more than 6 planes as I wanted to support higher levels of zoom with higher resolution meshes. More objects means more culling (since you don't have to send quite as much data to the videocard memory at any given time) so my method is a little harder to follow than Sebastians:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">generate_face</span><span>(
</span><span> </span><span style="color:#bf616a;">normal</span><span>: Vec3,
</span><span> </span><span style="color:#bf616a;">resolution</span><span>: </span><span style="color:#b48ead;">u32</span><span>,
</span><span> </span><span style="color:#bf616a;">x_offset</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#bf616a;">y_offset</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span>) -> Mesh {
</span><span> </span><span style="color:#b48ead;">let</span><span> axis_a = Vec3::new(normal.y, normal.z, normal.x); </span><span style="color:#65737e;">// Horizontal
</span><span> </span><span style="color:#b48ead;">let</span><span> axis_b = axis_a.</span><span style="color:#96b5b4;">cross</span><span>(normal); </span><span style="color:#65737e;">// Vertical
</span><span>
</span><span> </span><span style="color:#65737e;">// Create a vec of verticies and indicies
</span><span> </span><span style="color:#b48ead;">let mut</span><span> verticies: Vec<Vec3> = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> indicies: Vec<</span><span style="color:#b48ead;">u32</span><span>> = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> normals = Vec::new();
</span><span> </span><span style="color:#b48ead;">for</span><span> y in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">for</span><span> x in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">let</span><span> i = x + y * resolution;
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> percent = Vec2::new(x as </span><span style="color:#b48ead;">f32</span><span>, y as </span><span style="color:#b48ead;">f32</span><span>) / (resolution - </span><span style="color:#d08770;">1</span><span>) as </span><span style="color:#b48ead;">f32</span><span>;
</span><span> </span><span style="color:#b48ead;">let</span><span> point_on_unit_cube =
</span><span> normal + (percent.x - x_offset) * axis_a + (percent.y - y_offset) * axis_b;
</span><span> </span><span style="color:#b48ead;">let</span><span> point_coords: Coordinates = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>().</span><span style="color:#96b5b4;">into</span><span>();
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> normalized_point = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS</span><span>;
</span><span>
</span><span> verticies.</span><span style="color:#96b5b4;">push</span><span>(normalized_point);
</span><span>
</span><span> normals.</span><span style="color:#96b5b4;">push</span><span>(-point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>());
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> x != resolution - </span><span style="color:#d08770;">1 </span><span>&& y != resolution - </span><span style="color:#d08770;">1 </span><span>{
</span><span> </span><span style="color:#65737e;">// First triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Second triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + </span><span style="color:#d08770;">1</span><span>);
</span><span> }
</span><span> }
</span><span> }
</span><span> </span><span style="color:#b48ead;">let</span><span> indicies = mesh::Indices::</span><span style="color:#d08770;">U32</span><span>(indicies);
</span><span> </span><span style="color:#b48ead;">let mut</span><span> mesh = Mesh::new(PrimitiveTopology::TriangleList);
</span><span> mesh.</span><span style="color:#96b5b4;">set_indices</span><span>(Some(indicies));
</span><span> mesh.</span><span style="color:#96b5b4;">insert_attribute</span><span>(Mesh::</span><span style="color:#d08770;">ATTRIBUTE_POSITION</span><span>, verticies);
</span><span> mesh.</span><span style="color:#96b5b4;">insert_attribute</span><span>(Mesh::</span><span style="color:#d08770;">ATTRIBUTE_NORMAL</span><span>, normals);
</span><span> mesh.</span><span style="color:#96b5b4;">generate_tangents</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> mesh
</span><span>}
</span></code></pre>
<p>So, lets break this down a bit further:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">generate_face</span><span>(
</span><span> </span><span style="color:#bf616a;">normal</span><span>: Vec3,
</span><span> </span><span style="color:#bf616a;">resolution</span><span>: </span><span style="color:#b48ead;">u32</span><span>,
</span><span> </span><span style="color:#bf616a;">x_offset</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span> </span><span style="color:#bf616a;">y_offset</span><span>: </span><span style="color:#b48ead;">f32</span><span>,
</span><span>) -> Mesh {
</span></code></pre>
<p>Starting with the signature, we are going to generate a plane facing in a direction, the direction is determined by the <code>normal</code>. This will make more sense when we look a the wrapper function that calls this one.</p>
<p>Our <code>resolution</code> determines the number of vertices we will generate.</p>
<p>The <code>x_offset</code> and <code>y_offset</code> will determine the offset from the central vector. This might not be very intuitive, but it allows us to generate more than 1 plane per "side" of the cube (that we will then turn into a sphere.) Again, this will be clearer once we see the wrapper function.</p>
<p>And our output will be a <a href="https://docs.rs/bevy/latest/bevy/prelude/struct.Mesh.html">Mesh</a>.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> axis_a = Vec3::new(normal.y, normal.z, normal.x); </span><span style="color:#65737e;">// Horizontal
</span><span style="color:#b48ead;">let</span><span> axis_b = axis_a.</span><span style="color:#96b5b4;">cross</span><span>(normal); </span><span style="color:#65737e;">// Vertical
</span></code></pre>
<p>From our normal direction, we can derive the direction of our <code>x</code> and <code>y</code> axes of the plane. Fortunately this is very easy as Bevy provides some nice helper methods.</p>
<p>Then we initialize some things we will push various kinds of data into</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">// Create a vec of verticies and indicies
</span><span style="color:#b48ead;">let mut</span><span> verticies: Vec<Vec3> = Vec::new();
</span><span style="color:#b48ead;">let mut</span><span> indicies: Vec<</span><span style="color:#b48ead;">u32</span><span>> = Vec::new();
</span><span style="color:#b48ead;">let mut</span><span> normals = Vec::new();
</span></code></pre>
<p>And now it gets a little more interesting.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">for</span><span> y in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">for</span><span> x in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">let</span><span> i = x + y * resolution;
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> percent = Vec2::new(x as </span><span style="color:#b48ead;">f32</span><span>, y as </span><span style="color:#b48ead;">f32</span><span>) / (resolution - </span><span style="color:#d08770;">1</span><span>) as </span><span style="color:#b48ead;">f32</span><span>;
</span><span> </span><span style="color:#b48ead;">let</span><span> point_on_unit_cube =
</span><span> normal + (percent.x - x_offset) * axis_a + (percent.y - y_offset) * axis_b;
</span><span> </span><span style="color:#b48ead;">let</span><span> point_coords: Coordinates = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>().</span><span style="color:#96b5b4;">into</span><span>();
</span><span> </span><span style="color:#b48ead;">let</span><span> normalized_point = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS</span><span>;
</span><span>
</span><span> verticies.</span><span style="color:#96b5b4;">push</span><span>(normalized_point);
</span><span>
</span><span> normals.</span><span style="color:#96b5b4;">push</span><span>(-point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>());
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> x != resolution - </span><span style="color:#d08770;">1 </span><span>&& y != resolution - </span><span style="color:#d08770;">1 </span><span>{
</span><span> </span><span style="color:#65737e;">// First triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Second triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + </span><span style="color:#d08770;">1</span><span>);
</span><span> }
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<p>The first part is probably pretty intuitive, we have a loop for <code>y</code> and <code>x</code> in resolution. This means that a resolution of 100 will result in 100x100 verticies per face. Next we have this <code>i</code> value, which is an Index. An index for what you might ask? Don't get ahead of yourself, we are going to spend a lot of time talking about this index in a minute.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> percent = Vec2::new(x as </span><span style="color:#b48ead;">f32</span><span>, y as </span><span style="color:#b48ead;">f32</span><span>) / (resolution - </span><span style="color:#d08770;">1</span><span>) as </span><span style="color:#b48ead;">f32</span><span>;
</span></code></pre>
<p>We need to know how far along we are with each face, so this will keep track of that in a vec2 for the x and y axis respectively.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> point_on_unit_cube =
</span><span> normal + (percent.x - x_offset) * axis_a + (percent.y - y_offset) * axis_b;
</span><span style="color:#b48ead;">let</span><span> point_coords: Coordinates = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>().</span><span style="color:#96b5b4;">into</span><span>();
</span><span style="color:#b48ead;">let</span><span> normalized_point = point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>() * </span><span style="color:#d08770;">EARTH_RADIUS</span><span>;
</span></code></pre>
<p>Now we generate the actual xyz coordinates of our sphere. Now it may have seemed like we were going to generate a cube out of planes and then flatten them later, but in reality its a lot easier to do this in one step.</p>
<p>First we need to get the point on unit cube, this would be the pre-sphere point (if you used these coordinates for the verticies, it would actually be a cube with flat sides.) This is done by taking the normal and adding the normal plus our x and y percents, minus our x and y offset, times our <code>axis_a</code> and <code>axis_b</code> vectors.</p>
<p>Next, we take the resulting <a href="https://docs.rs/bevy/latest/bevy/math/struct.Vec3.html">Vec3</a> and <a href="https://docs.rs/bevy/latest/bevy/math/struct.Vec3.html#method.normalize">normalize</a> it. This is the step where our cube becomes a sphere. </p>
<p>Now that we have the normalized point, we can make the earth whatever radius we want, in this case I'm using a constant of <code>300.0</code>, but you can make it whatever radius you want.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>verticies.</span><span style="color:#96b5b4;">push</span><span>(normalized_point);
</span><span>normals.</span><span style="color:#96b5b4;">push</span><span>(-point_on_unit_cube.</span><span style="color:#96b5b4;">normalize</span><span>());
</span></code></pre>
<p>Now we need to push our normalized point to our verticies vec. Additionally, we will re-use our point_on_unit_cube variable to set the normal for this vertex correctly. This step probably isn't necessary; all of my normals were the correct way (we will talk about why in a second), but it is probably a good practice.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">if</span><span> x != resolution - </span><span style="color:#d08770;">1 </span><span>&& y != resolution - </span><span style="color:#d08770;">1 </span><span>{
</span><span> </span><span style="color:#65737e;">// First triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Second triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + </span><span style="color:#d08770;">1</span><span>);
</span><span>}
</span></code></pre>
<p>Now we need to unwrap this. In doing so, we will learn a fundemental truth of video-cards:</p>
<h1 id="everything-is-a-triangle">Everything Is a Triangle</h1>
<p>So we have a list of verticies already, and its probably rather intuitive that these verticies will make up the points of our mesh. But how do we draw the lines between each adjacent vertex in our plane? Heck, which vertexes do we draw lines between in the first place?</p>
<p>To answer this question, we have to talk about video cards. Video cards can only draw triangles <sup class="footnote-reference"><a href="#1">1</a></sup>. As a result, we need to construct our mesh from triangles. There are a lot of ways to do that, but we will be using Bevy's <a href="https://docs.rs/bevy/latest/bevy/render/mesh/enum.PrimitiveTopology.html#variant.TriangleList">PrimitiveTopology::TriangleList</a>.</p>
<p>This is one of the more primitive ways of constructing a mesh, but its also the closest to how the video card is actually going to render things which makes it very powerful (and often a bit less confusing for simpler shapes, like a plane.)</p>
<p>To generate this kind of mesh, we need two things (at a minium).</p>
<ul>
<li>A list of Vertex positions</li>
<li>A list of Indicies that defining triangles</li>
</ul>
<p>To explain this, lets start with a simple 2d example:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>
</span><span> (0, 0) (1, 0)
</span><span>
</span><span> ┌───────────────────┐
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> └───────────────────┘
</span><span>
</span><span> (0, 1) (1, 1)
</span><span>
</span></code></pre>
<p>Since we must create our mesh from triangles, we need two triangles to re-create this box. For example:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Triangle 1: [(0,0), (1,0), (0,1)]
</span><span>Triangle 2: [(0,1), (1,0), (1,1)]
</span></code></pre>
<p>However, this isn't going to work for our video card, because the order in which you specify the verticies is extremely important. In order to save a ton of memory and GPU cycles, most graphics libraries implement something called "backface culling" by default. This means they will only render one side of each triangle. So how do you tell the videocard which side of the face is the "front" vs the "back"?</p>
<p>According to the <a href="https://docs.rs/bevy/latest/bevy/prelude/struct.Mesh.html#use-with-standardmaterial">docs</a>:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Vertex winding order: by default, StandardMaterial.cull_mode is Some
</span><span>(Face::Back), which means that Bevy would only render the “front” of each
</span><span>triangle, which is the side of the triangle from where the vertices appear
</span><span>in a counter-clockwise order.
</span></code></pre>
<p>This is our answer, the winding order of the verticies tells the videocard which side of the triangle is the "front" vs "back." If you want the face facing you, you wind from the counter-clockwise direction when looking at it from the outside (Or the "normal" direction.)</p>
<p>So a valid set of triangles for our example could be:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>1: [(1,0), (0,0), (0,1)]
</span><span>2: [(1,1), (1,0), (0,1)]
</span></code></pre>
<p>But how do we specify that in Bevy? Now we need to return to the Index we glossed over earlier.</p>
<p>Remember I said before we needed two things: A list of Verticies and an index that defines triangles? Well now that we know they must be counter-clockwise, we are armed with all the information we need to go about making them.</p>
<p>So, our list of verticies could be:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>[(0,0), (0,1), (1,0), (1,1)]
</span></code></pre>
<p>And the indexes for our two triangles could be:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>[0, 1, 2, 0, 2, 1]
</span></code></pre>
<p>This would mean our two triangles are:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>1: [(1,0), (0,0), (0,1)]
</span><span>2: [(1,1), (1,0), (0,1)]
</span></code></pre>
<p>If you aren't sure you got that, read it again and try to let it sink in. Its not intuitive <em>at all</em>, so don't feel bad if doesn't make much sense at first.</p>
<p>Once you grok that, we can go back to the code that generates our indexes:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">if</span><span> x != resolution - </span><span style="color:#d08770;">1 </span><span>&& y != resolution - </span><span style="color:#d08770;">1 </span><span>{
</span><span> </span><span style="color:#65737e;">// First triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">// Second triangle
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + resolution + </span><span style="color:#d08770;">1</span><span>);
</span><span> indicies.</span><span style="color:#96b5b4;">push</span><span>(i + </span><span style="color:#d08770;">1</span><span>);
</span><span>}
</span></code></pre>
<p>Now, lets apply this to our 2d box example (which has a resolution of 2):</p>
<p>Starting at index 0, our first triangle indexes are:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>(i, i+1, i+1+1)
</span></code></pre>
<p>Or:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>(0, 1, 2)
</span></code></pre>
<p>And our second triangle indexes are:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>(i, i+1+1, i+1)
</span></code></pre>
<p>Or:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>(0, 2, 1)
</span></code></pre>
<p>Note that this strategy is entirely dependent on the way our loops are structured:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">for</span><span> y in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> </span><span style="color:#b48ead;">for</span><span> x in </span><span style="color:#d08770;">0</span><span>..(resolution) {
</span><span> }
</span><span>}
</span></code></pre>
<p>Because we are iterating over the y axis first and creating columns along the x axis, we need our index to look like:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>
</span><span> 0 2
</span><span>
</span><span> ┌───────────────────┐
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> │ │
</span><span> └───────────────────┘
</span><span>
</span><span> 1 3
</span><span>
</span></code></pre>
<p>And the last important bit is our if condition of <code>x != resolution - 1 && y != resolution - 1</code>. This ensures that we stop making triangles when we get to the end of a row or column.</p>
<p>Now, we need a function to wrap this one that will generate all sides of our cube-sphere:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub fn </span><span style="color:#8fa1b3;">generate_faces</span><span>(
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">commands</span><span>: Commands,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">meshes</span><span>: ResMut<Assets<Mesh>>,
</span><span> </span><span style="color:#b48ead;">mut </span><span style="color:#bf616a;">materials</span><span>: ResMut<Assets<StandardMaterial>>,
</span><span> </span><span style="color:#bf616a;">asset_server</span><span>: Res<AssetServer>,
</span><span>) {
</span><span> </span><span style="color:#b48ead;">let</span><span> faces = vec![
</span><span> Vec3::X,
</span><span> Vec3::</span><span style="color:#d08770;">NEG_X</span><span>,
</span><span> Vec3::Y,
</span><span> Vec3::</span><span style="color:#d08770;">NEG_Y</span><span>,
</span><span> Vec3::Z,
</span><span> Vec3::</span><span style="color:#d08770;">NEG_Z</span><span>,
</span><span> ];
</span><span>
</span><span> </span><span style="color:#b48ead;">let</span><span> offsets = vec![(</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>), (</span><span style="color:#d08770;">0.0</span><span>, </span><span style="color:#d08770;">1.0</span><span>), (</span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#d08770;">0.0</span><span>), (</span><span style="color:#d08770;">1.0</span><span>, </span><span style="color:#d08770;">1.0</span><span>)];
</span><span>
</span><span> </span><span style="color:#b48ead;">let mut</span><span> rng = rand::thread_rng();
</span><span>
</span><span> </span><span style="color:#b48ead;">for</span><span> direction in faces {
</span><span> </span><span style="color:#b48ead;">for</span><span> offset in &offsets {
</span><span> commands.</span><span style="color:#96b5b4;">spawn</span><span>((
</span><span> PbrBundle {
</span><span> mesh: meshes.</span><span style="color:#96b5b4;">add</span><span>(</span><span style="color:#96b5b4;">generate_face</span><span>(direction, </span><span style="color:#d08770;">16</span><span>, offset.</span><span style="color:#d08770;">0</span><span>, offset.</span><span style="color:#d08770;">1</span><span>, &rs)),
</span><span> material: materials.</span><span style="color:#96b5b4;">add</span><span>(StandardMaterial {
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> }),
</span><span> ..</span><span style="color:#96b5b4;">default</span><span>()
</span><span> }
</span><span> ));
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<p>This function will generate 24 faces, 4 per side of the "cube". The offsets and vectors hopefully make a bit more sense now.</p>
<p>The nice thing is, this method scales up to any resolution, so you can play around with varying polygon counts very easily.</p>
<p>And, here is what the end result looks like:</p>
<img src="../../images/plain_mesh_sphere.png" alt="A boring, untextured, sphere." />
<p>Not looking very interesting yet, is it? Don't worry, we will fix that soon! Stay tuned for Part 2.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>This is an Oversimplification. Video cards can draw all kinds of polygons in 2d. But in a 3d mesh, everything is always decomposed to triangles first. The chief reason being: 3 verticies is the exact number you need to uniquely specify a plane.</p>
</div>
Is Nix Worth The Hype?2023-09-23T00:00:00+00:002023-09-23T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/nixos-hype/<p>Sometimes something comes along that legitimately disrupts the software/IT industry. Some examples that come to mind are Ansible, Docker, Kubernetes, etc. But far more often, projects and products come along that make exaggerated claims of being revolutionary, only to die an unceremonious death.</p>
<p>At a certian point it becomes clear to everyone who the winners and the losers are. But by that time they are always so far along their journey of marketplace dominance that the job market is saturated by experts and anyone getting on board is on the laggard side of the adoption curve. </p>
<p>One thing is clear; <a href="https://nixos.org/">Nix</a> isn't there yet. But will it ever be? Is it worth your valuable time learning about it now?</p>
<h1 id="disambiguating-nix">Disambiguating Nix</h1>
<p>Its important to make a distinction between Nix: the package manager, and NixOS: the linux distribution that uses the Nix package manager. And then, to make things extra confusing, you also have Nix: the domain specific language. I'll forgive you being confused because I'll be referring to the package manager and the language interchangeably. </p>
<p>In truth, it won't affect the conclusions of this post because what I want to evaluate the viability of the ecosystem of Nix, which includes all of the aforementioned components. How do these things all add up? Do they really make managing software and systems easier? Or is it just another useless layer of abstraction?</p>
<h1 id="the-promise-of-panacea">The Promise of Panacea</h1>
<p>Looking back to when I learned ansible, I remember being filled with its promise. I was still working for a company that largely managed servers like pets (because everyone did in those days except for the rare hyper-scale exceptions.) I remember thinking that Ansible was going to revolutionize the way I provisoined machines, and in all honesty it did. It solved a lot of problems, but more importantly, it changed my mindset on how large herds of computers should be managed.</p>
<p>Ansible had a lot of shortcomings though, for instance, while it tried very hard to be declarative it was still very much operating on an imperative CLI or against imperative APIs. While most of the core plugins were idempotent (you can run them multiple times and it won't mess things up if the system is already in the correct state), there was no functional way of forcing plugin developers to write idempotent plugins. As a result, as you left the officially blessed core plugins, you entered a wild west where side effects and unexpected behavior were more and more likely. </p>
<p>Ansible was a very important step in improving the way provisioning worked, but it wasn't a panacea. </p>
<p>Docker was another huge step forward for the industry. It has very minimal scripts (Dockerfiles) that allow you to build an image that can be deployed on systems that contained their own environments. It solved a lot of traditional problems that came from running in the shared library environment of traditional Linux. It made traditionally hard-to-deploy applications much easier.</p>
<p>But this introduced a new class of problems. From tried and true build scripts to fragile Dockerfiles that were often vulnerable to bitrot. Additionally relying on a user defined label instead of a hash by default (don't even get me started on the <code>latest</code> tag) led to a lot of potential security issues, and performance problems and bugs that were difficult to debug (why are half of my application servers crashing when they are all running the same version?) Additionally IO and Network issues are commonplace, and vulnerability fixes (meltdown and spectre specifically) resulted in a lot of strange performance bugs inside of containers.</p>
<p>Docker was another huge step, but again, far from a panacea.</p>
<h1 id="what-problems-does-nix-actually-solve">What Problems Does Nix Actually Solve?</h1>
<p>The Nix ecosystem is capable of deterministically and repeatably building software, and entire operating systems (including their configuration). It puts guard rails (isolated sandboxes and source control) in place to ensure that the same inputs <em>always</em> deliver the same outputs. In short, the same inputs will always give you (bit for bit) the same output. </p>
<p>Why is this important?</p>
<p>Firstly, it allows you to build software fearlessly. The same inputs will always produce the same outputs, so if you build something 10 different times on 10 different computers (assuming they run the same architecture), you get the same bit-for-bit result 10 times, even if you build the last one several years apart. This allows you to do fun things like building binaries, uploading them to a shared cache, and then transparently substituting the output when the input hash matches. This means your install process is identical to your build process, which gives you immense flexibility. You don't <em>need</em> a CI/CD pipeline to build artifacts and then worry about deploying those artifacts to your servers. You can absolutely have a CI/CD pipeline that warms up a cache so you don't build binaries on your application servers, but this is completely optional (you should still have one though, at least for tests.) </p>
<p>Secondly, it allows you to deterministically build operating systems. If you deploy 10 different servers from the same source, you will get the exact same server with the exact same behavior 10 times. Even if you build them several years apart. This completely eliminates a lot of the problems associated with imperative management schemes (for example, deploying something imperatively on a OS and then deploying something on a slightly newer OS which can result in wildly different performance due to library or OS tuning differences.) In other words, if you fix a problem once, you have inherently fixed it everywhere without having to worry about how to idempotently deploy that fix. Every single time you rebuild the system to deploy a change, you get a brand new fresh operating system (or as Nix calls them, a generation.)</p>
<p>Putting all this together, what it means is that you can have a single repository (or several interconnected ones) that completely describes your server and build environment. And if you need to rebuild your entire environment a year from now, you will get the exact same environment.</p>
<p>Additionally, there are other benefits as well which I'm glossing over, such as the ability to deterministically roll back to a previous state in the case of a failed upgrade, and having an immense amount of flexibility regarding your deployment style. Being able to push (deploy-rs) and pull (rebuild against a Nix Flake in a git repo) in the same repo without configuration changes makes for a very nice workflow of allowing rapid iteration while also being extremely scalable.</p>
<p>I'd also like to stress that deterministically building entire operating systems without any kind of heavy handed image cloning <em>is a novel thing</em>. It hasn't really been done before, and it can't really be done without making some of the ground-up design choices that NixOS made (such as abandoning <a href="https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html">FHS</a> in favor of the nix store.)</p>
<h1 id="what-price-do-you-pay-for-the-upsides">What Price Do You Pay For The Upsides?</h1>
<p>I'm gonna spoil the conclusion, Nix isn't a panacea either.</p>
<p>Nix as a language can be a real drag. It isn't because it is designed badly. It solves a very specific set of problems, and as a result has a specific set of conventions that may not be obvious or ergonomic to even seasoned developers/operations folk. This is doubly true if they have never used a purely functional language before.</p>
<p>To compound these issues, Nix doesn't really lend itself to toy examples or programs. Its a language to describe configurations, so the only thing you really can do with it is make packages and operating systems. As a result, the minimal viable example for most things is as complex as building a package. The simpler examples don't really teach you anything, and the complexity ramp is quite drastic in many cases.</p>
<p>The last thing any software engineer wants is to be instructed to learn a language they don't care about, and most software engineers don't appear give two hoots about packaging. So twisting the arm of any reasonably large organization, especially one with multiple development teams, to start packaging with Nix is going to be quite difficult unless there are a lot of developers already inclined to explore it.</p>
<p>This is especially true for NixOS adoption specifically, because excluding strategies like running Docker images or VMs on a NixOS box, you need your software and all of its dependencies packaged in Nix in order to deploy and orchestrate things.</p>
<p>I really can't overstate how big this hurdle is.</p>
<h1 id="what-is-missing">What Is Missing?</h1>
<p>My opinion is, there are three big things needed to bridge the gap:</p>
<h2 id="documentation">Documentation</h2>
<p>The state of Nix documentation is pretty bad.</p>
<p>The two main sources are the <a href="https://nixos.org/manual/nix/stable/">Nix Manual</a> and the <a href="https://nixos.org/manual/nixos/stable/">NixOS manual</a>.</p>
<p>There is an ideal ratio of time spent writing code to time spent writing documentation. No one knows what it actually is, but most people intuitively know when it isn't followed. However in the case of Nix it isn't so clear. There is, in fact, a <em>lot</em> of documentation. Its just that most of it isn't easy to follow. The documentation reads extremely terse, almost like an RFC or standards document. It seems to assume that the reader already groks the theoretical models of Nix and are looking for specific information. </p>
<p>For a reference manual, this would be fine. But for a guide meant to introduce new users, its just going to make people feel stupid and by extension piss them off. It lacks any kind of narrative. It spends an awful lot of time <em>describing</em> how things work, and no time <em>showing</em> how it works. The lack of examples in what should be the first stop for new users is a huge problem. </p>
<h2 id="standardized-practices">Standardized Practices</h2>
<p>One of Nix's biggest strengths is its flexibility. You can do a lot of creative things with packaging and systems configuration that are straight up not possible in alternative systems. However, this has a huge drawback: A total lack of standardization between different users and organizations.</p>
<p><a href="https://nixos.wiki/wiki/Flakes">Flakes</a> have made some strides in resolving this. Having a standardized input and output set that can be consumed by other Flakes does make collaborative Nix usage a lot more predictable. But anything made before the advent of flakes has a bit of "wild west" feel to it; you never really know how it works until you go read the source.</p>
<h2 id="minified-examples">Minified Examples</h2>
<p>Searching for minified examples of how to do X is difficult when it comes to Nix. I'm honestly not sure why this is, but it is something that I've heard countless colleagues complain about. I myself have run into this dozens of time. How do you include a configuration module in a Flake? How do you change the source of an application using its overrides attribute? How do you combine the result of two build outputs? None of these things are hard in Nix, but figuring out how to do them in the first place is difficult. It usually requires stumbling around in the source code of Nixpkgs or random Flakes until you finally run into what you are trying to do and see how someone else did it. Very few examples exist alone in isolation. </p>
<h1 id="the-state-of-modern-nix-and-its-users">The State of Modern Nix and Its Users</h1>
<p>Nix has been around since 2003, which is ages in the technology world. This might lead a lot of people to believe that its missed the boat on mass adoption. However, my personal feelings is that before the advent of Nix Flakes, Nix was a novel concept with no practical use outside of a Hobbyist OS. </p>
<p>For the record Nix Flakes were introduced in 2021 as experimental, and have not yet been marked as stable. However, I think you'd be foolish to try and introduce Nix to a new organization without Flakes. Flakes make Nix composable, and a declaratively built operating system using deterministically constructed software composed from multiple federated repositories is the secret sauce of Nix as an ecosystem. It allows you to define your <em>entire</em> infrastructure in Git (which isn't novel), and be able to rebuild it over and over again and get the exact same results every time (which is novel.)</p>
<p>So, considering the relative new-ness of flakes, I'd say modern nix is still very much in the early adopter quadrant of the adoption curve, and it is far from dead in the water.</p>
<p>While I give "old" Nix a lot of crap, Nixpkgs (the monorepo) has a <em>ton</em> of software in it. And it all stays pretty well up to date, at least all the less obscure stuff. You are not often going to run into the situation where a package available for Ubuntu or Centos isn't available in Nix. And since the build process for packages doesn't bit rot, keeping packages up to date is often a lot easier than it is with traditional distributions as the amount of work a maintainer needs to exert is far greatly reduced once the initial package is defined.</p>
<p>Functionally re-creating a basic server in Nix using common open source technologies is easy bordering on trivial, once you figure out the <code>nixos-rebuild</code> workflow. It doesn't require you to write much Nix at all. This is true up until the point where you start packaging your own software. At that point, you have to package your software (as well as any missing dependencies.) In organizations with lots of in-house software this can be a lot of work</p>
<h1 id="is-it-worth-using-now">Is It Worth Using Now?</h1>
<p>I have firsthand knowledge of a few companies that either already adopted or are well on their way to adopting Nix as an ecosystem. All of them have encountered some of the problems listed above, and in most cases there does need to be some organic interest within the engineering organization to get efforts off of the ground. </p>
<p>But, with that being said, the benefits are clear. Package reproducibility and all of its many positive side effects seem to legitimately eliminate a large swathe of problems that relate to fragile build processes. Using Nix to replace your traditional build pipeline in my experience typically means that you get to trust it. Which isn't something I'd typically say of a more traditional pipeline, especially if Groovy scripts or Dockerfiles are involved. </p>
<p>Furthermore, using Nix to manage your systems has the benefit of your systems being far more consistent than they could be with other deployment mechanisms. At least, not without investing far more effort than would be reasonable. And when you fix a problem, it typically stays fixed. </p>
<p>If Nix has a theme I imagine it is something along the lines of: "investing a bit more effort in your up-front setup can save you a lot more effort chasing problems later". For me, the answer is clear; the tradeoff is well worth it.</p>
Nix Powered Dev Environments: Rust + Bevy2023-06-20T00:00:00+00:002023-06-20T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/nix-flake-rust-bevy/<p>As with many things Nix, the result was simple and reliably repeatable, but getting there took a lot of throwing stuff at the wall. Currently my small allotment of game development time is focused on Rust, and more specifically Bevy; so if you aren't interested in that combination this post may not be much help to you.</p>
<p>If you want to just take my word for it and download a template, I'll save you some time:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>nix flake new --template github:graysonhead/nix-templates#rust-bevy ./new_directory_here
</span></code></pre>
<p>This should get you a mostly out of the box "hello world" 3d scene running on bevy 0.10. At least, it should work out of the box on NixOS. For non NixOS, keep reading.</p>
<p>For posterity, here are the two most important parts:</p>
<p><code>flake.nix</code></p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">inputs </span><span>= {
</span><span> </span><span style="color:#d08770;">flake-utils</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:numtide/flake-utils</span><span>";
</span><span> </span><span style="color:#d08770;">naersk</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:nix-community/naersk</span><span>";
</span><span> </span><span style="color:#d08770;">rust-overlay</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:oxalica/rust-overlay</span><span>";
</span><span> </span><span style="color:#d08770;">nixpkgs</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:NixOS/nixpkgs/nixpkgs-unstable</span><span>";
</span><span> };
</span><span>
</span><span> </span><span style="color:#d08770;">outputs </span><span>= </span><span style="color:#8fa1b3;">{ </span><span>self, flake-utils, naersk, nixpkgs, rust-overlay </span><span style="color:#8fa1b3;">}</span><span>:
</span><span> </span><span style="color:#bf616a;">flake-utils</span><span>.</span><span style="color:#bf616a;">lib</span><span>.</span><span style="color:#bf616a;">eachDefaultSystem </span><span>(system:
</span><span> </span><span style="color:#b48ead;">let
</span><span> </span><span style="color:#d08770;">overlays </span><span>= [ (</span><span style="color:#96b5b4;">import </span><span style="color:#bf616a;">rust-overlay</span><span>) ];
</span><span> </span><span style="color:#d08770;">pkgs </span><span>= (</span><span style="color:#96b5b4;">import </span><span style="color:#bf616a;">nixpkgs</span><span>) {
</span><span> </span><span style="color:#b48ead;">inherit </span><span style="color:#d08770;">system overlays</span><span>;
</span><span> };
</span><span> </span><span style="color:#d08770;">naersk' </span><span>= </span><span style="color:#bf616a;">pkgs</span><span>.</span><span style="color:#bf616a;">callPackage naersk </span><span>{ };
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span> </span><span style="color:#bf616a;">vulkan-loader
</span><span> </span><span style="color:#bf616a;">xorg</span><span>.</span><span style="color:#bf616a;">libXcursor
</span><span> </span><span style="color:#bf616a;">xorg</span><span>.</span><span style="color:#bf616a;">libXi
</span><span> </span><span style="color:#bf616a;">xorg</span><span>.</span><span style="color:#bf616a;">libXrandr
</span><span> </span><span style="color:#bf616a;">alsa-lib
</span><span> </span><span style="color:#bf616a;">udev
</span><span> </span><span style="color:#bf616a;">pkg-config
</span><span> ];
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span> </span><span style="color:#bf616a;">libxkbcommon
</span><span> (</span><span style="color:#bf616a;">rust-bin</span><span>.</span><span style="color:#bf616a;">selectLatestNightlyWith
</span><span> (toolchain: </span><span style="color:#bf616a;">toolchain</span><span>.</span><span style="color:#bf616a;">default</span><span>.</span><span style="color:#bf616a;">override </span><span>{
</span><span> </span><span style="color:#d08770;">extensions </span><span>= [ "</span><span style="color:#a3be8c;">rust-src</span><span>" "</span><span style="color:#a3be8c;">clippy</span><span>" ];
</span><span> }))
</span><span> ];
</span><span> </span><span style="color:#d08770;">all_deps </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span> </span><span style="color:#bf616a;">cargo-flamegraph
</span><span> </span><span style="color:#bf616a;">cargo-expand
</span><span> </span><span style="color:#bf616a;">nixpkgs-fmt
</span><span> </span><span style="color:#bf616a;">cmake
</span><span> ] ++ </span><span style="color:#bf616a;">buildInputs </span><span>++ </span><span style="color:#bf616a;">nativeBuildInputs</span><span>;
</span><span> </span><span style="color:#b48ead;">in
</span><span> </span><span style="color:#b48ead;">rec </span><span>{
</span><span> </span><span style="color:#65737e;"># For `nix build` & `nix run`:
</span><span> </span><span style="color:#d08770;">defaultPackage </span><span>= </span><span style="color:#bf616a;">packages</span><span>.</span><span style="color:#bf616a;">bevy_template</span><span>;
</span><span> </span><span style="color:#d08770;">packages </span><span>= </span><span style="color:#b48ead;">rec </span><span>{
</span><span> </span><span style="color:#d08770;">bevy_template </span><span>= </span><span style="color:#bf616a;">naersk'</span><span>.</span><span style="color:#bf616a;">buildPackage </span><span>{
</span><span> </span><span style="color:#d08770;">src </span><span>= </span><span style="color:#a3be8c;">./.</span><span>;
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#bf616a;">nativeBuildInputs</span><span>;
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#bf616a;">buildInputs</span><span>;
</span><span> </span><span style="color:#d08770;">postInstall </span><span>= ''
</span><span style="color:#a3be8c;"> cp -r assets $out/bin/
</span><span style="color:#a3be8c;"> </span><span>'';
</span><span> </span><span style="color:#65737e;"># Disables dynamic linking when building with Nix
</span><span> </span><span style="color:#d08770;">cargoBuildOptions </span><span>= x: </span><span style="color:#bf616a;">x </span><span>++ [ "</span><span style="color:#a3be8c;">--no-default-features</span><span>" ];
</span><span> };
</span><span> };
</span><span>
</span><span> </span><span style="color:#65737e;"># For `nix develop`:
</span><span> </span><span style="color:#d08770;">devShell </span><span>= </span><span style="color:#bf616a;">pkgs</span><span>.</span><span style="color:#bf616a;">mkShell </span><span>{
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#bf616a;">all_deps</span><span>;
</span><span> </span><span style="color:#d08770;">shellHook </span><span>= ''
</span><span style="color:#a3be8c;"> export CARGO_MANIFEST_DIR=$(pwd)
</span><span style="color:#a3be8c;"> export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:</span><span style="font-style:italic;color:#ab7967;">${</span><span style="font-style:italic;color:#bf616a;">pkgs</span><span style="font-style:italic;color:#c0c5ce;">.</span><span style="font-style:italic;color:#bf616a;">lib</span><span style="font-style:italic;color:#c0c5ce;">.</span><span style="font-style:italic;color:#bf616a;">makeLibraryPath all_deps</span><span style="font-style:italic;color:#ab7967;">}</span><span style="color:#a3be8c;">"
</span><span style="color:#a3be8c;"> </span><span>'';
</span><span> };
</span><span> }
</span><span> );
</span><span>}
</span><span>
</span></code></pre>
<p><code>cargo.toml</code></p>
<pre data-lang="toml" style="background-color:#2b303b;color:#c0c5ce;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[package]
</span><span style="color:#bf616a;">name </span><span>= "</span><span style="color:#a3be8c;">bevy_template</span><span>"
</span><span style="color:#bf616a;">version </span><span>= "</span><span style="color:#a3be8c;">0.1.0</span><span>"
</span><span style="color:#bf616a;">edition </span><span>= "</span><span style="color:#a3be8c;">2021</span><span>"
</span><span>
</span><span style="color:#65737e;"># See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
</span><span>
</span><span>[dependencies]
</span><span style="color:#bf616a;">bevy </span><span>= { </span><span style="color:#bf616a;">package </span><span>= "</span><span style="color:#a3be8c;">bevy</span><span>", </span><span style="color:#bf616a;">version </span><span>= "</span><span style="color:#a3be8c;">0.13.2</span><span>" }
</span><span>
</span><span>[profile.dev]
</span><span style="color:#bf616a;">opt-level </span><span>= </span><span style="color:#d08770;">1
</span><span>
</span><span>[profile.dev.package."</span><span style="color:#a3be8c;">*</span><span>"]
</span><span style="color:#bf616a;">opt-level </span><span>= </span><span style="color:#d08770;">3
</span><span>
</span><span>[features]
</span><span style="color:#bf616a;">default </span><span>= ["</span><span style="color:#a3be8c;">bevy/dynamic_linking</span><span>"]
</span><span>
</span></code></pre>
<p>Going first over the <code>cargo.toml</code>, you will note that I'm following the advice of the <a href="https://bevy-cheatbook.github.io/pitfalls/performance.html">Unofficial Bevy Cheat Book</a>. </p>
<pre data-lang="toml" style="background-color:#2b303b;color:#c0c5ce;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[profile.dev]
</span><span style="color:#bf616a;">opt-level </span><span>= </span><span style="color:#d08770;">1
</span><span>
</span><span>[profile.dev.package."</span><span style="color:#a3be8c;">*</span><span>"]
</span><span style="color:#bf616a;">opt-level </span><span>= </span><span style="color:#d08770;">3
</span></code></pre>
<p>By setting the profile.dev to opt-level 1 and dev.package."*" to opt-level 3, you get a great compromise of compile times (as the code you are changing is at an optimization level of 1), but all the dependencies you are using (such as bevy itself) is at optimization level 3. In some cases, you may still run into performance issues, in which case you should increase the profile.dev opt level, or alternatively just compile in release mode.</p>
<p>Speaking of which, <code>nix build</code> will always compile in release mode.</p>
<p>Now, if you really want to get the compile times down, you will enable the <code>dynamic_linking</code> (formerly known as <code>dynamic</code> in pre 0.10 releases of Bevy). However, this prevents <code>nix build</code> or <code>nix run</code> from working, because dynamic linking in Nix is...odd. At least in regards to Rust.</p>
<p>So, to combat this, I made this rather counter-intuitive addition to the Cargo manifest:</p>
<pre data-lang="toml" style="background-color:#2b303b;color:#c0c5ce;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[dependencies]
</span><span style="color:#bf616a;">bevy </span><span>= { </span><span style="color:#bf616a;">package </span><span>= "</span><span style="color:#a3be8c;">bevy</span><span>", </span><span style="color:#bf616a;">version </span><span>= "</span><span style="color:#a3be8c;">0.10.1</span><span>"}
</span><span>
</span><span>[features]
</span><span style="color:#bf616a;">default </span><span>= ["</span><span style="color:#a3be8c;">bevy/dynamic_linking</span><span>"]
</span></code></pre>
<p>So, by default if you run <code>cargo build</code> or <code>cargo run</code>, you will get very fast compile times thanks to dynamic linking, however once you build the derivation in release mode using nix, it won't use dynamic linking (and will actually work as a result.)</p>
<p>The <code>cargoBuildOptions</code> line in the flake governs that behavior:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span style="color:#bf616a;">packages </span><span style="background-color:#bf616a;color:#2b303b;">=</span><span> </span><span style="color:#b48ead;">rec </span><span>{
</span><span> </span><span style="color:#d08770;">bevy_template </span><span>= </span><span style="color:#bf616a;">naersk'</span><span>.</span><span style="color:#bf616a;">buildPackage </span><span>{
</span><span> </span><span style="color:#d08770;">src </span><span>= </span><span style="color:#a3be8c;">./.</span><span>;
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#bf616a;">nativeBuildInputs</span><span>;
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#bf616a;">buildInputs</span><span>;
</span><span> </span><span style="color:#d08770;">postInstall </span><span>= ''
</span><span style="color:#a3be8c;"> cp -r assets $out/bin/
</span><span style="color:#a3be8c;"> </span><span>'';
</span><span> </span><span style="color:#65737e;"># Disables dynamic linking when building with Nix
</span><span> </span><span style="color:#d08770;">cargoBuildOptions </span><span>= x: </span><span style="color:#bf616a;">x </span><span>++ [ "</span><span style="color:#a3be8c;">--no-default-features</span><span>" ];
</span><span> };
</span><span>}</span><span style="background-color:#bf616a;color:#2b303b;">;</span><span>
</span></code></pre>
<p>In addition, the <code>postInstall</code> line ensures that any game assets are copied to the correct directory. </p>
<p>So, you get the best of both worlds more or less. Consistent development environments using Flakes, fast compile times using cargo, and statically linked outputs using nix.</p>
<h1 id="non-nixos-systems">Non NixOS Systems</h1>
<p>If you want to run this example using Nix on a non-NixOS system, you will need an extra step. Since your graphics drivers are most likely not available in the Nix store, you will need to employ a wrapper to make them available. </p>
<p>The best one I've found is <a href="https://github.com/guibou/nixGL">NixGL</a>. </p>
<p>Use it like this:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>nix run --impure github:guibou/nixGL -- nix run
</span></code></pre>
<p>Or, if your Nix install doesn't have Non-free packages enabled by default:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>export NIXPKGS_ALLOW_UNFREE=1 && nix run --impure github:guibou/nixGL -- nix run
</span></code></pre>
Nix Powered Dev Environments: Rust2023-04-24T00:00:00+00:002023-04-24T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/sane-nix-flake-rust/<p>Nix is not just a language, package manager, or operating system, but an ecosystem with remarkable capabilities. </p>
<p>This tutorial is targeted for someone who is getting their feet wet with Nix, using the Nix
package manager on Linux or MacOS. But, it also demonstrates a development workflow
that can be used by other crazy people like me that daily drive NixOS.</p>
<p>I started using Nix about a year and a half ago, and since then I've
gone deep down the rabbit hole. To the point where my entire home setup runs
NixOS primarily. Having a consistent experience across all
of my computers is a wonderful thing, and one of the best parts is
having truly 100% isolated development environments. </p>
<p>While bootstrapping these is a bit more work, (due to an intentional complete
lack of development tools in my system path), having near
guaranteed isolation between projects is
quite an incredible thing.</p>
<p>The core components of this development workflow center around <a href="https://nixos.wiki/wiki/Flakes">Nix Flakes</a> and <a href="https://direnv.net/">Direnv</a>.</p>
<h1 id="flake-templates">Flake Templates</h1>
<p>Due to the increased initial cost of setting up a new project, its almost mandatory to
create standardized Flake templates to save yourself time when kickstarting a new project.</p>
<p>I maintain a repo of <a href="https://github.com/graysonhead/nix-templates">Flake Templates</a> customized for my use, but they are MIT licensed so feel free to use them.</p>
<p>Today we are going to explore the Rust templates that I still actively use.</p>
<h1 id="simple-naersk-nixpkgs">Simple: Naersk + Nixpkgs</h1>
<p>Here is a simple, but useful, Rust Flake example:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#65737e;"># This is our input set, it contains the channels we will construct the flake from.
</span><span> </span><span style="color:#d08770;">inputs </span><span>= {
</span><span> </span><span style="color:#65737e;"># Flake-utils allows us to easily support multiple architectures.
</span><span> </span><span style="color:#d08770;">flake-utils</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:numtide/flake-utils</span><span>";
</span><span> </span><span style="color:#65737e;"># Naersk is a zero-configuration zero-codegen solution to packaging Rust.
</span><span> </span><span style="color:#d08770;">naersk</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:nix-community/naersk</span><span>";
</span><span> </span><span style="color:#65737e;"># Nixpkgs is the main package repo for Nix, we will use it to bring in all of our
</span><span> </span><span style="color:#65737e;"># libraries and tools.
</span><span> </span><span style="color:#d08770;">nixpkgs</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:NixOS/nixpkgs/nixpkgs-unstable</span><span>";
</span><span> };
</span><span>
</span><span> </span><span style="color:#65737e;"># This is the output set, Nix and other Flakes will be able to consume the attributes
</span><span> </span><span style="color:#65737e;"># in the output set.
</span><span> </span><span style="color:#d08770;">outputs </span><span>= </span><span style="color:#8fa1b3;">{ </span><span>self, flake-utils, naersk, nixpkgs </span><span style="color:#8fa1b3;">}</span><span>:
</span><span> </span><span style="color:#65737e;"># We wrap the entire output set in this flake-utils function, which builds the flake
</span><span> </span><span style="color:#65737e;"># for each architecture type supported by nix.
</span><span> </span><span style="color:#bf616a;">flake-utils</span><span>.</span><span style="color:#bf616a;">lib</span><span>.</span><span style="color:#bf616a;">eachDefaultSystem </span><span>(system:
</span><span> </span><span style="color:#b48ead;">let
</span><span> </span><span style="color:#65737e;"># This sets up nixpkgs, where we will pull our dependencies from
</span><span> </span><span style="color:#d08770;">pkgs </span><span>= (</span><span style="color:#96b5b4;">import </span><span style="color:#bf616a;">nixpkgs</span><span>) {
</span><span> </span><span style="color:#65737e;"># You can insert overlays here by calling `inherit system overlays;`
</span><span> </span><span style="color:#b48ead;">inherit </span><span style="color:#d08770;">system</span><span>;
</span><span> };
</span><span>
</span><span> </span><span style="color:#65737e;"># This sets up naersk, which we will use later.
</span><span> </span><span style="color:#d08770;">naersk' </span><span>= </span><span style="color:#bf616a;">pkgs</span><span>.</span><span style="color:#bf616a;">callPackage naersk </span><span>{ };
</span><span>
</span><span> </span><span style="color:#65737e;"># Here we can add non-rust dependencies that our program requires *at run time*.
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span>
</span><span> ];
</span><span>
</span><span> </span><span style="color:#65737e;"># here we can add non-rust dependencies that our program requires *at build time*.
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span>
</span><span> ];
</span><span> </span><span style="color:#b48ead;">in
</span><span> </span><span style="color:#b48ead;">rec </span><span>{
</span><span> </span><span style="color:#65737e;"># Build this with `nix build`, run it with `nix run`
</span><span> </span><span style="color:#d08770;">defaultPackage </span><span>= </span><span style="color:#bf616a;">packages</span><span>.</span><span style="color:#bf616a;">app</span><span>;
</span><span> </span><span style="color:#d08770;">packages </span><span>=
</span><span> {
</span><span> </span><span style="color:#d08770;">app </span><span>= </span><span style="color:#bf616a;">naersk'</span><span>.</span><span style="color:#bf616a;">buildPackage </span><span>{
</span><span> </span><span style="color:#65737e;"># Naersk will look for a `Cargo.toml` in this directory
</span><span> </span><span style="color:#d08770;">src </span><span>= </span><span style="color:#a3be8c;">./.</span><span>;
</span><span> </span><span style="color:#65737e;"># Our buildinputs from above are specified here
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#bf616a;">nativeBuildInputs</span><span>;
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#bf616a;">buildInputs</span><span>;
</span><span> };
</span><span> </span><span style="color:#d08770;">contianer </span><span>= </span><span style="color:#bf616a;">pkgs</span><span>.</span><span style="color:#bf616a;">dockerTools</span><span>.</span><span style="color:#bf616a;">buildImage
</span><span> {
</span><span> </span><span style="color:#d08770;">name </span><span>= "</span><span style="color:#a3be8c;">app</span><span>";
</span><span> </span><span style="color:#d08770;">config </span><span>= {
</span><span> </span><span style="color:#d08770;">entrypoint </span><span>= [ "</span><span style="font-style:italic;color:#ab7967;">${</span><span style="font-style:italic;color:#bf616a;">packages</span><span style="font-style:italic;color:#c0c5ce;">.</span><span style="font-style:italic;color:#bf616a;">app</span><span style="font-style:italic;color:#ab7967;">}</span><span style="color:#a3be8c;">/bin/app</span><span>" ];
</span><span> };
</span><span> };
</span><span> };
</span><span>
</span><span> </span><span style="color:#65737e;"># This will be entered by direnv, or by manually running `nix shell`. This ensures
</span><span> </span><span style="color:#65737e;"># that our development environment will have all the correct tools at the correct
</span><span> </span><span style="color:#65737e;"># version for this project.
</span><span> </span><span style="color:#d08770;">devShell </span><span>= </span><span style="color:#bf616a;">pkgs</span><span>.</span><span style="color:#bf616a;">mkShell </span><span>{
</span><span> </span><span style="color:#65737e;"># Here we add any tools that we want in our dev-shell but aren't required to build
</span><span> </span><span style="color:#65737e;"># our application.
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>;
</span><span> [
</span><span> </span><span style="color:#bf616a;">nixpkgs-fmt
</span><span> </span><span style="color:#bf616a;">cmake
</span><span> </span><span style="color:#bf616a;">rustc
</span><span> </span><span style="color:#bf616a;">cargo
</span><span> </span><span style="color:#bf616a;">clippy
</span><span> ] ++ </span><span style="color:#bf616a;">buildInputs </span><span>++ </span><span style="color:#bf616a;">nativeBuildInputs</span><span>;
</span><span> </span><span style="color:#65737e;"># The above line merges our buildInputs into the devshell, so we have them when
</span><span> </span><span style="color:#65737e;"># using cargo tools from inside our devshell.
</span><span> };
</span><span> }
</span><span> );
</span><span>}
</span><span>
</span></code></pre>
<p>You can setup this repo locally with this command:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>nix flake new --template github:graysonhead/nix-templates#rust-naersk ./changeme
</span></code></pre>
<p>This is the base template I use for any simple rust project I use that will be based off of
nixpgks and not require a nightly (or otherwise specific) build of Rust. Its a good starting
point, and it is already pretty flexible. For instance, if you are building something that
requires openssl you only need to change one line to add it:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span> </span><span style="color:#bf616a;">openssl
</span><span> ];
</span><span>}
</span></code></pre>
<p>Using it is also simple. First, you shoud familiarize yourself with the output of <code>nix flake show</code>:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ nix flake show
</span><span>git+file:///home/grayson/RustProjects/naersk-template
</span><span>├───defaultPackage
</span><span>│ ├───aarch64-darwin: package 'naersk-template-0.1.0'
</span><span>│ ├───aarch64-linux: package 'naersk-template-0.1.0'
</span><span>│ ├───x86_64-darwin: package 'naersk-template-0.1.0'
</span><span>│ └───x86_64-linux: package 'naersk-template-0.1.0'
</span><span>└───devShell
</span><span> ├───aarch64-darwin: development environment 'nix-shell'
</span><span> ├───aarch64-linux: development environment 'nix-shell'
</span><span> ├───x86_64-darwin: development environment 'nix-shell'
</span><span> └───x86_64-linux: development environment 'nix-shell'
</span><span>
</span></code></pre>
<p>Here we can see a tree of the output set (the set that the nix flake returns) in a more visually appealing form. This command is
extremely useful on more complex flakes with dozens or even hundreds of outputs, but here it gives us a good idea of what is going on.</p>
<p>First, notice that both our <code>defaultPackage</code> and <code>devShell</code> have outputs for each architecture already. This is a result of the
<code>flake-utils.lib.eachDefaultSystem</code> function that we wrapped our output set in. It ensures that the flake is repeated for each system type.</p>
<p>As a result, our flake will work on Linux and MacOS on both x86 and ARM systems. Best of all, we got that functionality practically for free!</p>
<p>Since we defined a <code>defaultPackage</code>, we can build the application by running <code>nix build</code>, or run it with <code>nix run</code>. You can enter the development
shell with <code>nix develop</code>, which will allow you to compile your program with native tools such as <code>cargo</code> as well as <code>clippy</code>.</p>
<p>To target a specific output, you can simply trace down the tree to your desired output, and call <code>nix build</code> or <code>nix run</code> like this:
<code>nix run .#defaultPackage.x86_64-linux</code>, or whatever arch you are on.</p>
<h1 id="containers">Containers</h1>
<p>Another nice thing about Nix flakes, is that you can easily build
docker containers without needing a fragile dockerfile. For instance, you can
modify the output set and add a seperate output for building a container image:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">packages </span><span>=
</span><span> {
</span><span> </span><span style="color:#d08770;">app </span><span>= </span><span style="color:#bf616a;">naersk'</span><span>.</span><span style="color:#bf616a;">buildPackage </span><span>{
</span><span> </span><span style="color:#65737e;"># Naersk will look for a `Cargo.toml` in this directory
</span><span> </span><span style="color:#d08770;">src </span><span>= </span><span style="color:#a3be8c;">./.</span><span>;
</span><span> </span><span style="color:#65737e;"># Our buildinputs from above are specified here
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#bf616a;">nativeBuildInputs</span><span>;
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#bf616a;">buildInputs</span><span>;
</span><span> };
</span><span> </span><span style="color:#d08770;">contianer </span><span>= </span><span style="color:#bf616a;">pkgs</span><span>.</span><span style="color:#bf616a;">dockerTools</span><span>.</span><span style="color:#bf616a;">buildImage
</span><span> {
</span><span> </span><span style="color:#d08770;">name </span><span>= "</span><span style="color:#a3be8c;">app</span><span>";
</span><span> </span><span style="color:#d08770;">config </span><span>= {
</span><span> </span><span style="color:#d08770;">entrypoint </span><span>= [ "</span><span style="font-style:italic;color:#ab7967;">${</span><span style="font-style:italic;color:#bf616a;">packages</span><span style="font-style:italic;color:#c0c5ce;">.</span><span style="font-style:italic;color:#bf616a;">app</span><span style="font-style:italic;color:#ab7967;">}</span><span style="color:#a3be8c;">/bin/app</span><span>" ];
</span><span> };
</span><span> };
</span><span> };
</span><span>}
</span></code></pre>
<p>This allows you to build a container image by running <code>nix build .#container</code>, which
can be loaded into docker via <code>docker load -i ./result</code>. </p>
<h1 id="multi-workspace-projects">Multi-Workspace Projects</h1>
<p>Flakes also allow you to utilize workspaces as well, although the process for doing that
in Naersk isn't exactly straightforward. </p>
<p>To build a project in a workspace, you can add this to your package output definition:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">example-workspaced-app </span><span>= </span><span style="color:#bf616a;">naersk'</span><span>.</span><span style="color:#bf616a;">buildPackage </span><span>{
</span><span> </span><span style="color:#d08770;">name </span><span>= "</span><span style="color:#a3be8c;">example-workspace</span><span>";
</span><span> </span><span style="color:#d08770;">src </span><span>= </span><span style="color:#a3be8c;">./.</span><span>;
</span><span> </span><span style="color:#d08770;">cargoBuildOptions </span><span>= x: </span><span style="color:#bf616a;">x </span><span>++ [ "</span><span style="color:#a3be8c;">-p</span><span>" "</span><span style="color:#a3be8c;">example-workspace</span><span>" ];
</span><span> };
</span><span>}
</span></code></pre>
<p>This assumes that your workspace (with its own <code>Cargo.toml</code> file) is located in <code>./example-workspace</code>.</p>
<p>Note that you will also need to ensure that your main <code>Cargo.toml</code> file has an entry
for this workspace, like this:</p>
<pre data-lang="toml" style="background-color:#2b303b;color:#c0c5ce;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[workspace]
</span><span style="color:#bf616a;">members </span><span>= [
</span><span> "</span><span style="color:#a3be8c;">example-workspace</span><span>"
</span><span>]
</span></code></pre>
<p>Cargo technically doesn't <em>need</em> this entry to function, but Naersk does, and you
won't be able to build or run anything in that workspace without it.</p>
<h1 id="nightly-rust">Nightly Rust</h1>
<p>If you want to use nightly rust (or a version seperate from Nixpkgs), you can use the <code>oxalica/rust-overlay</code>. </p>
<p>Here is a full example:</p>
<pre data-lang="nix" style="background-color:#2b303b;color:#c0c5ce;" class="language-nix "><code class="language-nix" data-lang="nix"><span>{
</span><span> </span><span style="color:#d08770;">inputs </span><span>= {
</span><span> </span><span style="color:#d08770;">flake-utils</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:numtide/flake-utils</span><span>";
</span><span> </span><span style="color:#d08770;">naersk</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:nix-community/naersk</span><span>";
</span><span> </span><span style="color:#d08770;">rust-overlay</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:oxalica/rust-overlay</span><span>";
</span><span> </span><span style="color:#d08770;">nixpkgs</span><span>.</span><span style="color:#d08770;">url </span><span>= "</span><span style="color:#a3be8c;">github:NixOS/nixpkgs/nixpkgs-unstable</span><span>";
</span><span> };
</span><span>
</span><span> </span><span style="color:#d08770;">outputs </span><span>= </span><span style="color:#8fa1b3;">{ </span><span>self, flake-utils, naersk, nixpkgs, rust-overlay </span><span style="color:#8fa1b3;">}</span><span>:
</span><span> </span><span style="color:#bf616a;">flake-utils</span><span>.</span><span style="color:#bf616a;">lib</span><span>.</span><span style="color:#bf616a;">eachDefaultSystem </span><span>(system:
</span><span> </span><span style="color:#b48ead;">let
</span><span> </span><span style="color:#d08770;">overlays </span><span>= [ (</span><span style="color:#96b5b4;">import </span><span style="color:#bf616a;">rust-overlay</span><span>) ];
</span><span> </span><span style="color:#d08770;">pkgs </span><span>= (</span><span style="color:#96b5b4;">import </span><span style="color:#bf616a;">nixpkgs</span><span>) {
</span><span> </span><span style="color:#b48ead;">inherit </span><span style="color:#d08770;">system overlays</span><span>;
</span><span> };
</span><span>
</span><span> </span><span style="color:#d08770;">naersk' </span><span>= </span><span style="color:#bf616a;">pkgs</span><span>.</span><span style="color:#bf616a;">callPackage naersk </span><span>{ };
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span> ];
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span> </span><span style="color:#65737e;"># This sets up the rust suite, automatically selecting the latest nightly version
</span><span> (</span><span style="color:#bf616a;">rust-bin</span><span>.</span><span style="color:#bf616a;">selectLatestNightlyWith
</span><span> (toolchain: </span><span style="color:#bf616a;">toolchain</span><span>.</span><span style="color:#bf616a;">default</span><span>.</span><span style="color:#bf616a;">override </span><span>{
</span><span> </span><span style="color:#d08770;">extensions </span><span>= [ "</span><span style="color:#a3be8c;">rust-src</span><span>" "</span><span style="color:#a3be8c;">clippy</span><span>" ];
</span><span> }))
</span><span> ];
</span><span> </span><span style="color:#b48ead;">in
</span><span> </span><span style="color:#b48ead;">rec </span><span>{
</span><span> </span><span style="color:#65737e;"># For `nix build` & `nix run`:
</span><span> </span><span style="color:#d08770;">defaultPackage </span><span>= </span><span style="color:#bf616a;">packages</span><span>.</span><span style="color:#bf616a;">naersk-nightly</span><span>;
</span><span> </span><span style="color:#d08770;">packages </span><span>= </span><span style="color:#b48ead;">rec </span><span>{
</span><span> </span><span style="color:#d08770;">naersk-nightly </span><span>= </span><span style="color:#bf616a;">naersk'</span><span>.</span><span style="color:#bf616a;">buildPackage </span><span>{
</span><span> </span><span style="color:#d08770;">src </span><span>= </span><span style="color:#a3be8c;">./.</span><span>;
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#bf616a;">nativeBuildInputs</span><span>;
</span><span> </span><span style="color:#d08770;">buildInputs </span><span>= </span><span style="color:#bf616a;">buildInputs</span><span>;
</span><span> };
</span><span> };
</span><span>
</span><span> </span><span style="color:#65737e;"># For `nix develop`:
</span><span> </span><span style="color:#d08770;">devShell </span><span>= </span><span style="color:#bf616a;">pkgs</span><span>.</span><span style="color:#bf616a;">mkShell </span><span>{
</span><span> </span><span style="color:#d08770;">nativeBuildInputs </span><span>= </span><span style="color:#b48ead;">with </span><span style="color:#bf616a;">pkgs</span><span>; [
</span><span> </span><span style="color:#bf616a;">cargo-expand
</span><span> </span><span style="color:#bf616a;">nixpkgs-fmt
</span><span> </span><span style="color:#bf616a;">cmake
</span><span> ] ++ </span><span style="color:#bf616a;">buildInputs </span><span>++ </span><span style="color:#bf616a;">nativeBuildInputs</span><span>;
</span><span> };
</span><span> }
</span><span> );
</span><span>}
</span><span>
</span></code></pre>
<h1 id="making-development-shells-automatic">Making Development Shells Automatic</h1>
<p>Using <a href="https://direnv.net/">direnv</a>, and <a href="https://github.com/nix-community/nix-direnv">nix-direnv</a>, you can make the activation of your development
environment automatic. Add an <code>.envrc</code> file in the root of your repo with the line:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>use flake
</span></code></pre>
<p>And then run the command <code>direnv allow</code> from this directory. Your development shell will now be activated any time you change directory to that location.</p>
<p>This means that your entire environment will automatically switch whenever you change between projects, meaning you don't have to deal with rustup and
toolchains whenever you switch between a nightly and stable Rust project.</p>
<p>You can also get IDE/terminal editor plugins that allow you to use this environment as well, so that your <code>rust-analyzer</code> version is appropriate to the version
of Rust you are developing on. I use <code>cab404.vscode-direnv</code> for vscode, and <a href="https://github.com/direnv/direnv.vim">direnv-vim</a> for vim.</p>
<h1 id="bonus-round-nix-party-tricks">Bonus Round: Nix Party Tricks</h1>
<p>Lets look at some other fun stuff you can trivially do now that you have this set up. </p>
<p>Run a program from a remote git repo without permanently installing it in your profile or system path:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ nix run github:nixos/nixpkgs/nixpkgs-unstable#hello
</span></code></pre>
<p>You can also cross-compile it to a different architecture (requires for <code>boot.binfmt.emulatedSystems</code> to be set appropriately on NixOS):</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ nix build github:nixos/nixpkgs/nixpkgs-unstable#legacyPackages.aarch64-linux.hello
</span><span>$ file result/bin/hello
</span><span>result/bin/hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/rjpx52ch4508wrq8wjf5nnbsc6pr3158-glibc-2.37-8/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped
</span></code></pre>
<p>Need to distribute your program via RPM or Deb? In most cases, this is trivial:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ nix bundle --bundler github:NixOS/bundlers#toDEB .#defaultPackage.x86_64-linux
</span><span>$ nix bundle --bundler github:NixOS/bundlers#toRPM .#defaultPackage.x86_64-linux
</span></code></pre>
Async TCP in Rust Without Shooting Yourself in the Foot2023-04-18T00:00:00+00:002023-04-18T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/rust-tcp/<p>As someone who came to Rust from high level languages, lower level networking was something
that I was very excited to work with in Rust. While Python's TCP and UDP implementations are
great, the speed of Python often felt very constraining when writing services that needed
to move lots of data quickly and concurrently. </p>
<p>I was glad to discover that working with the async implementations of TCP and UDP sockets in
Rust using <a href="https://tokio.rs/">Tokio</a> was quite intuitive, for the most part. While getting traffic flowing was
easy, getting traffic flowing <em>reliably</em> required some steps beyond the boilerplate in the
documentation. So, let my lessons be your lessons, lets send some segments!</p>
<h2 id="tcp-basics">TCP Basics</h2>
<p>Frankly, TCP is a complicated way of getting data from point A to point B. There are a lot
of steps to set up, send data, acknowledge the receipt of data, and terminate the connection.
Fortunately, the OS handles most of this for us. The sockets our program will interact
with seem a lot like magical data gateways, and that is the point of TCP in a nutshell. </p>
<p>Compared to User Datagram Protocl (UDP), TCP has a lot of advantages. Namely:</p>
<ul>
<li>Delivery order of data is guaranteed</li>
<li>Re-transmission of data is automatic if not acknowledged by the other side</li>
<li>Flow and Congestion control</li>
<li>You can easily send blocs of data larger than your network's Maximum Transmission Unit (MTU)</li>
</ul>
<p>But, there is a price to pay:</p>
<ul>
<li>Programs using TCP are often more complex. This is primarily due to keeping track of more connection states</li>
<li>TCP itself has some overhead, a system using many TCP sessions may use more memory than one using UDP for
the same purpose</li>
<li>Due to complex flow handling and congestion control algorithms, sometimes its harder to troubleshoot TCP performance issues</li>
</ul>
<p>Generally, TCP is useful if you need guaranteed delivery of data in a specific order. It can also reduce the complexity of
programs that need a lot of bidirectional connections between a single server and multiple clients. It also can make it
easier to send large blocs of data larger than your MTU, but there are no shortage of libraries that make this just as easy with
UDP.</p>
<h1 id="getting-started-streams-and-listeners">Getting Started: Streams and Listeners</h1>
<p>Lets get some connections going. Lets start a project and add a dependency for our async runtime, Tokio:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>cargo new rust-tcp-tutorial
</span><span>cargo add tokio --features socket2 --features macros --features rt-multi-thread --features net --features sync --features io-util
</span></code></pre>
<p>Next, we will add two rust files in <code>/src/bin/</code>:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">// src/bin/server.rs
</span><span style="color:#b48ead;">use </span><span>tokio::net::TcpListener;
</span><span>
</span><span>#[</span><span style="color:#bf616a;">tokio</span><span>::</span><span style="color:#bf616a;">main</span><span>]
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> bind_addr = "</span><span style="color:#a3be8c;">0.0.0.0:1234</span><span>";
</span><span> </span><span style="color:#96b5b4;">listen</span><span>(bind_addr).await;
</span><span>}
</span><span>
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">listen</span><span>(</span><span style="color:#bf616a;">bind_addr</span><span>: &</span><span style="color:#b48ead;">str</span><span>) {
</span><span> </span><span style="color:#b48ead;">let</span><span> listener = TcpListener::bind(bind_addr).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> </span><span style="color:#b48ead;">loop </span><span>{
</span><span> </span><span style="color:#b48ead;">let </span><span>(stream, _) = listener.</span><span style="color:#96b5b4;">accept</span><span>().await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> println!("</span><span style="color:#a3be8c;">Connection from </span><span style="color:#d08770;">{}</span><span>", stream.</span><span style="color:#96b5b4;">peer_addr</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>());
</span><span> }
</span><span>}
</span></code></pre>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">// src/bin/client.rs
</span><span style="color:#b48ead;">use </span><span>tokio::net::TcpStream;
</span><span>
</span><span>#[</span><span style="color:#bf616a;">tokio</span><span>::</span><span style="color:#bf616a;">main</span><span>]
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> server_addr = "</span><span style="color:#a3be8c;">127.0.0.1:1234</span><span>";
</span><span> </span><span style="color:#b48ead;">let</span><span> stream: TcpStream = TcpStream::connect(server_addr).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> println!("</span><span style="color:#a3be8c;">Connected to </span><span style="color:#d08770;">{}</span><span>", stream.</span><span style="color:#96b5b4;">peer_addr</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>());
</span><span>}
</span><span>
</span></code></pre>
<p>Our first example doesn't do much useful. If you run the two programs at the same time, you
will see that all they do is initiate a connection.</p>
<p>You can run any rust file in <code>/src/bin</code> like this:<br />
<code>cargo run --bin server</code><br />
<code>cargo run --bin client</code></p>
<p>On the server, you will get the output:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Connection from 127.0.0.1:55586
</span></code></pre>
<p>And on the client:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Connected to 127.0.0.1:1234
</span></code></pre>
<p>Taking a closer look at what is happening thus far: On the server,
we are binding to a socket address, which is represented by a bind IP and port. A bind address
is a mechanism to inform the host operating system upon which interfaces this listener will be bound.</p>
<p>For instance, say we have a system with the following Interface layout:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>eth0 10.1.1.15
</span><span>eth1 172.16.0.25
</span><span>lo 127.0.0.1
</span></code></pre>
<p>If the bind address were set to <code>127.0.0.1:1234</code>, any request to <code>eth0</code> or <code>eth1</code> would fail (usually with
a connection refused error), but a request sent to the local loopback address of <code>127.0.0.1</code> with a
destination port of <code>1234</code> would succeed.</p>
<p>A bind address of of <code>10.5.5.15:1234</code> would allow connections to <code>eth0</code> but not <code>eth1</code> or the local loopback.</p>
<p>If you want to bind to all available interfaces, you can use an bind IP of <code>0.0.0.0:1234</code>.</p>
<p>The call to <code>TcpListener::bind()</code>, unsurprisingly returns a <a href="https://docs.rs/tokio/latest/tokio/net/struct.TcpListener.html">TcpListener</a>. </p>
<p>Once a client connects, we can call <code>listener.accept()</code> to get a <a href="https://docs.rs/tokio/latest/tokio/net/struct.TcpStream.html">TcpStream</a>
for this connection. This is the mechanism that allows you to send and receive data to and from this client.</p>
<p>On the client side, you will notice that <code>TcpStream::connect()</code> of course also gives us a <code>TcpStream</code>. This is
convenient as there can be a lot of commonality in the way clients and servers interact.</p>
<p>So, to sum up, a TCP Server binds to a port, which returns a Listener. These Listeners then give us Streams, and
streams are what we will use in the next section to actually send and receive data. </p>
<h2 id="using-streams">Using Streams</h2>
<p>Before getting into some examples, I want to point out that the <a href="https://docs.rs/tokio/latest">Tokio docs</a> are a wonderful resource, and you
will see the skeleton of the examples in a lot of production code. But, they quite minimal, and they don't go over (or at least don't explain)
some of the more common pitfalls you may run into. The following not-so-minimal example will show a lot of this behavior.</p>
<p>Extending our previous code, lets add relay functionality to the server:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">// src/bin/server.rs
</span><span style="color:#b48ead;">use </span><span>std::{error::Error, io};
</span><span style="color:#b48ead;">use </span><span>tokio::{
</span><span> io::{AsyncReadExt, Interest},
</span><span> net::{TcpListener, TcpStream},
</span><span>};
</span><span>
</span><span>#[</span><span style="color:#bf616a;">tokio</span><span>::</span><span style="color:#bf616a;">main</span><span>]
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> bind_addr = "</span><span style="color:#a3be8c;">0.0.0.0:1234</span><span>";
</span><span> </span><span style="color:#96b5b4;">listen</span><span>(bind_addr).await;
</span><span>}
</span><span>
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">listen</span><span>(</span><span style="color:#bf616a;">bind_addr</span><span>: &</span><span style="color:#b48ead;">str</span><span>) {
</span><span> </span><span style="color:#b48ead;">let</span><span> listener = TcpListener::bind(bind_addr).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> </span><span style="color:#b48ead;">loop </span><span>{
</span><span> </span><span style="color:#b48ead;">let </span><span>(stream, _) = listener.</span><span style="color:#96b5b4;">accept</span><span>().await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> tokio::spawn(async </span><span style="color:#b48ead;">move </span><span>{
</span><span> </span><span style="color:#96b5b4;">handle_stream</span><span>(stream).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> });
</span><span> }
</span><span>}
</span><span>
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">handle_stream</span><span>(</span><span style="color:#bf616a;">stream</span><span>: TcpStream) -> Result<(), Box<dyn Error>> {
</span><span> println!("</span><span style="color:#a3be8c;">Connection from </span><span style="color:#d08770;">{}</span><span>", stream.</span><span style="color:#96b5b4;">peer_addr</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>());
</span><span> </span><span style="color:#b48ead;">let mut</span><span> reply_queue: Vec<Vec<</span><span style="color:#b48ead;">u8</span><span>>> = Vec::new();
</span><span> </span><span style="color:#b48ead;">let mut</span><span> buf: [</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">1024</span><span>];
</span><span> </span><span style="color:#b48ead;">loop </span><span>{
</span><span> </span><span style="color:#b48ead;">let</span><span> ready = stream
</span><span> .</span><span style="color:#96b5b4;">ready</span><span>(Interest::</span><span style="color:#d08770;">READABLE </span><span>| Interest::</span><span style="color:#d08770;">WRITABLE</span><span>)
</span><span> .await?;
</span><span> </span><span style="color:#b48ead;">if</span><span> ready.</span><span style="color:#96b5b4;">is_readable</span><span>() {
</span><span> buf = [</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#d08770;">1024</span><span>];
</span><span> </span><span style="color:#b48ead;">match</span><span> stream.</span><span style="color:#96b5b4;">try_read</span><span>(&</span><span style="color:#b48ead;">mut</span><span> buf) {
</span><span> Ok(n) => {
</span><span> println!("</span><span style="color:#a3be8c;">read </span><span style="color:#d08770;">{}</span><span style="color:#a3be8c;"> bytes</span><span>", n);
</span><span> </span><span style="color:#b48ead;">let mut</span><span> result_buffer: Vec<</span><span style="color:#b48ead;">u8</span><span>> = Vec::with_capacity(n);
</span><span> buf.</span><span style="color:#96b5b4;">take</span><span>(n as </span><span style="color:#b48ead;">u64</span><span>).</span><span style="color:#96b5b4;">read_to_end</span><span>(&</span><span style="color:#b48ead;">mut</span><span> result_buffer).await?;
</span><span> reply_queue.</span><span style="color:#96b5b4;">push</span><span>(buf.</span><span style="color:#96b5b4;">to_vec</span><span>());
</span><span> }
</span><span> Err(</span><span style="color:#b48ead;">ref</span><span> e) </span><span style="color:#b48ead;">if</span><span> e.</span><span style="color:#96b5b4;">kind</span><span>() == io::ErrorKind::WouldBlock => {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span> Err(e) => {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(e.</span><span style="color:#96b5b4;">into</span><span>());
</span><span> }
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">if</span><span> ready.</span><span style="color:#96b5b4;">is_writable</span><span>() {
</span><span> </span><span style="color:#b48ead;">if let </span><span>Some(msg) = reply_queue.</span><span style="color:#96b5b4;">pop</span><span>() {
</span><span> </span><span style="color:#b48ead;">match</span><span> stream.</span><span style="color:#96b5b4;">try_write</span><span>(&msg) {
</span><span> Ok(n) => {
</span><span> println!("</span><span style="color:#a3be8c;">Wrote </span><span style="color:#d08770;">{}</span><span style="color:#a3be8c;"> bytes</span><span>", n);
</span><span> }
</span><span> Err(</span><span style="color:#b48ead;">ref</span><span> e) </span><span style="color:#b48ead;">if</span><span> e.</span><span style="color:#96b5b4;">kind</span><span>() == io::ErrorKind::WouldBlock => {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span> Err(e) => {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(e.</span><span style="color:#96b5b4;">into</span><span>());
</span><span> }
</span><span> }
</span><span> }
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#65737e;">// src/bin/client.rs
</span><span style="color:#b48ead;">use </span><span>std::error::Error;
</span><span style="color:#b48ead;">use </span><span>std::io;
</span><span style="color:#b48ead;">use </span><span>tokio::io::AsyncReadExt;
</span><span style="color:#b48ead;">use </span><span>tokio::net::TcpStream;
</span><span>
</span><span>#[</span><span style="color:#bf616a;">tokio</span><span>::</span><span style="color:#bf616a;">main</span><span>]
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">main</span><span>() {
</span><span> </span><span style="color:#b48ead;">let</span><span> server_addr = "</span><span style="color:#a3be8c;">127.0.0.1:1234</span><span>";
</span><span> </span><span style="color:#b48ead;">let</span><span> stream: TcpStream = TcpStream::connect(server_addr).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> println!("</span><span style="color:#a3be8c;">Connected to </span><span style="color:#d08770;">{}</span><span>", stream.</span><span style="color:#96b5b4;">peer_addr</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>());
</span><span> </span><span style="color:#96b5b4;">send_request</span><span>(stream).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span>}
</span><span>
</span><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">send_request</span><span>(</span><span style="color:#bf616a;">stream</span><span>: TcpStream) -> Result<(), Box<dyn Error>> {
</span><span> </span><span style="color:#b48ead;">loop </span><span>{
</span><span> stream.</span><span style="color:#96b5b4;">writable</span><span>().await?;
</span><span> </span><span style="color:#b48ead;">match</span><span> stream.</span><span style="color:#96b5b4;">try_write</span><span>(</span><span style="color:#b48ead;">b</span><span>"</span><span style="color:#a3be8c;">Hello!</span><span>") {
</span><span> Ok(n) => {
</span><span> println!("</span><span style="color:#a3be8c;">Wrote </span><span style="color:#d08770;">{}</span><span style="color:#a3be8c;"> bytes</span><span>", n);
</span><span> </span><span style="color:#b48ead;">break</span><span>;
</span><span> }
</span><span> Err(</span><span style="color:#b48ead;">ref</span><span> e) </span><span style="color:#b48ead;">if</span><span> e.</span><span style="color:#96b5b4;">kind</span><span>() == io::ErrorKind::WouldBlock => {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span> Err(e) => {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(e.</span><span style="color:#96b5b4;">into</span><span>());
</span><span> }
</span><span> }
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">let mut</span><span> buf: [</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">4096</span><span>];
</span><span> </span><span style="color:#b48ead;">loop </span><span>{
</span><span> stream.</span><span style="color:#96b5b4;">readable</span><span>().await?;
</span><span> buf = [</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#d08770;">4096</span><span>];
</span><span> </span><span style="color:#b48ead;">match</span><span> stream.</span><span style="color:#96b5b4;">try_read</span><span>(&</span><span style="color:#b48ead;">mut</span><span> buf) {
</span><span> Ok(n) => {
</span><span> </span><span style="color:#b48ead;">let mut</span><span> vec = Vec::with_capacity(n);
</span><span> buf.</span><span style="color:#96b5b4;">take</span><span>(n as </span><span style="color:#b48ead;">u64</span><span>).</span><span style="color:#96b5b4;">read_to_end</span><span>(&</span><span style="color:#b48ead;">mut</span><span> vec).await?;
</span><span> </span><span style="color:#b48ead;">let</span><span> s = String::from_utf8(buf.</span><span style="color:#96b5b4;">to_vec</span><span>()).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> println!("</span><span style="color:#a3be8c;">Got reply from host: </span><span style="color:#d08770;">{}</span><span>", s);
</span><span> </span><span style="color:#b48ead;">break</span><span>;
</span><span> }
</span><span> Err(</span><span style="color:#b48ead;">ref</span><span> e) </span><span style="color:#b48ead;">if</span><span> e.</span><span style="color:#96b5b4;">kind</span><span>() == io::ErrorKind::WouldBlock => {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span> Err(e) => </span><span style="color:#b48ead;">return </span><span>Err(e.</span><span style="color:#96b5b4;">into</span><span>()),
</span><span> }
</span><span> }
</span><span>
</span><span> Ok(())
</span><span>}
</span></code></pre>
<p>Lets go over some of the more important parts of these changes. </p>
<p>The client is quite a naive example, as it very deterministically sends a request and then waits for a reply, but we
do see the start of two patterns you will see throughout the rest of this tutorial. Specifically, the <code>try_read</code> and <code>try_write</code>
match blocks.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">loop </span><span>{
</span><span> stream.</span><span style="color:#96b5b4;">writable</span><span>().await?;
</span><span> </span><span style="color:#b48ead;">match</span><span> stream.</span><span style="color:#96b5b4;">try_write</span><span>(</span><span style="color:#b48ead;">b</span><span>"</span><span style="color:#a3be8c;">Hello!</span><span>") {
</span><span> Ok(n) => {
</span><span> println!("</span><span style="color:#a3be8c;">Wrote </span><span style="color:#d08770;">{}</span><span style="color:#a3be8c;"> bytes</span><span>", n);
</span><span> </span><span style="color:#b48ead;">break</span><span>;
</span><span> }
</span><span> Err(</span><span style="color:#b48ead;">ref</span><span> e) </span><span style="color:#b48ead;">if</span><span> e.</span><span style="color:#96b5b4;">kind</span><span>() == io::ErrorKind::WouldBlock => {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span> Err(e) => {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(e.</span><span style="color:#96b5b4;">into</span><span>());
</span><span> }
</span><span> }
</span><span>}
</span></code></pre>
<p>So first, we check to see if the stream is actually ready to be written to. It often can't, for various reasons. And
sometimes it thinks it is, but it actually isn't. Notice that this method is async and thus is <code>await</code>ed. </p>
<p>There is a lot of really cool logic happening in this one line. On Linux operating systems, when Tokio
created the TcpStream, it registered an <a href="https://en.wikipedia.org/wiki/Epoll">epoll</a> waiter for the stream. When we
await on the <code>writeable</code> method, Tokio puts the task to sleep until the Kernel informs us either that the stream is
ready to be written to, or that data has been received
and is ready to be read on the stream. What this means is, this task will consume practically no system resources
until there is data for it to process. Pretty neat, huh?</p>
<p>Next, we match the result of the <code>try_write</code> method. Now, this isn't async, but <code>try_write</code> will ensure that the call
doesn't block on IO unnecessarily. However, this creates a quandry, as sometimes the <code>writable</code> method gives us false
positives because of weird messy kernel programming reasons. As a result, we need to be prepared to deal with that, and
we deal with it by matching on the <code>WouldBlock</code> error type. If we get this Error, we restart the loop and await it again.
Eventually, we will enter the loop on a true positive, and be able to write our data. </p>
<p>While this arrangement may sound complex, it can reduce the busy time of programs doing network IO <em>significantly</em>.</p>
<p>Reading response uses mostly the same pattern, but with more logic to handle copying data from buffers:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">loop </span><span>{
</span><span> </span><span style="color:#b48ead;">let mut</span><span> buf: [</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">4096</span><span>];
</span><span> stream.</span><span style="color:#96b5b4;">readable</span><span>().await?;
</span><span> buf = [</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#d08770;">4096</span><span>];
</span><span> </span><span style="color:#b48ead;">match</span><span> stream.</span><span style="color:#96b5b4;">try_read</span><span>(&</span><span style="color:#b48ead;">mut</span><span> buf) {
</span><span> Ok(n) => {
</span><span> </span><span style="color:#b48ead;">let mut</span><span> vec = Vec::with_capacity(n);
</span><span> buf.</span><span style="color:#96b5b4;">take</span><span>(n as </span><span style="color:#b48ead;">u64</span><span>).</span><span style="color:#96b5b4;">read_to_end</span><span>(&</span><span style="color:#b48ead;">mut</span><span> vec).await?;
</span><span> </span><span style="color:#b48ead;">let</span><span> s = String::from_utf8(buf.</span><span style="color:#96b5b4;">to_vec</span><span>()).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> println!("</span><span style="color:#a3be8c;">Got reply from host: </span><span style="color:#d08770;">{}</span><span>", s);
</span><span> </span><span style="color:#b48ead;">break</span><span>;
</span><span> }
</span><span> Err(</span><span style="color:#b48ead;">ref</span><span> e) </span><span style="color:#b48ead;">if</span><span> e.</span><span style="color:#96b5b4;">kind</span><span>() == io::ErrorKind::WouldBlock => {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span> Err(e) => </span><span style="color:#b48ead;">return </span><span>Err(e.</span><span style="color:#96b5b4;">into</span><span>()),
</span><span> }
</span><span> }
</span></code></pre>
<p>Now, for the server. Up first, our listen function is now spawning a Tokio Task on each new stream.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>async </span><span style="color:#b48ead;">fn </span><span style="color:#8fa1b3;">listen</span><span>(</span><span style="color:#bf616a;">bind_addr</span><span>: &</span><span style="color:#b48ead;">str</span><span>) {
</span><span> </span><span style="color:#b48ead;">let</span><span> listener = TcpListener::bind(bind_addr).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> </span><span style="color:#b48ead;">loop </span><span>{
</span><span> </span><span style="color:#b48ead;">let </span><span>(stream, _) = listener.</span><span style="color:#96b5b4;">accept</span><span>().await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> tokio::spawn(async </span><span style="color:#b48ead;">move </span><span>{
</span><span> </span><span style="color:#96b5b4;">handle_stream</span><span>(stream).await.</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span> });
</span><span> }
</span><span>}
</span></code></pre>
<p>Keep in mind, Tokio Tasks are not threads. Thread count is determined by the Tokio runtime. Tasks will
run on a thread until they <code>await</code>, at which point another task will usually take its place. So don't feel bad about
spinning up a bunch of tasks to handle individual client connections. Unless you are doing blocking io operations in them,
in which case you should feel bad.</p>
<p>Notice how we are spawning a task for each stream, and running the <code>handle_stream</code> function inside of it. While not
always necessary, this is a very common pattern for applications that want to process traffic for multiple clients
simultaneously. This also takes advantage of async IO, as the program will never block while waiting for messages
from hosts that haven't sent any.</p>
<p>And now, lets break the <code>handle_stream</code> function into a few distinct sections:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>println!("</span><span style="color:#a3be8c;">Connection from </span><span style="color:#d08770;">{}</span><span>", stream.</span><span style="color:#96b5b4;">peer_addr</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>());
</span><span style="color:#b48ead;">let mut</span><span> reply_queue: Vec<Vec<</span><span style="color:#b48ead;">u8</span><span>>> = Vec::new();
</span><span style="color:#b48ead;">let mut</span><span> buf: [</span><span style="color:#b48ead;">u8</span><span>; </span><span style="color:#d08770;">1024</span><span>];
</span></code></pre>
<p>In this "preamble", we are setting up a mutable <code>reply_queue</code> which we will use to store messages that we want to
relay back to the client. Since each client's stream exists within its own Tokio task running this function, this
isolates communication per client by default, which is a nice aspect of this pattern and lends itself well to serving
many clients simultaneously.</p>
<p>Now, getting into the loop:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span> </span><span style="color:#b48ead;">let</span><span> ready = stream
</span><span> .</span><span style="color:#96b5b4;">ready</span><span>(Interest::</span><span style="color:#d08770;">READABLE </span><span>| Interest::</span><span style="color:#d08770;">WRITABLE</span><span>)
</span><span> .await?;
</span></code></pre>
<p>This method returns a <a href="https://docs.rs/tokio/latest/tokio/io/struct.Ready.html">Ready</a> object describing the current
state of the Stream. This allows you to handle one write operation, or one read operation per loop. This also ensures that
your writes don't block your reads, your reads don't block your writes. Since it is <code>await</code>ed, it also ensures that the
Task will sleep while it has no ability to perform IO.</p>
<p>There will be a lot of cases in more complex programs where you need to split your <code>TcpStream</code> into a <code>WriteHalf</code> and a <code>ReadHalf</code>, but that is
a topic for another day.</p>
<p>Now, lets run the code. But first make sure you know where your <code>ctrl + c</code> keys are:</p>
<p>Client:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>Connected to 127.0.0.1:1234
</span><span>Wrote 6 bytes
</span><span>Got reply from host: Hello!
</span></code></pre>
<p>Server:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>read 6 bytes
</span><span>Wrote 6 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>Wrote 0 bytes
</span><span>read 0 bytes
</span><span>...
</span></code></pre>
<h2 id="whats-up-with-the-zero-byte-buffer">Whats Up With the Zero Byte Buffer?</h2>
<p>Well, that is something...</p>
<p>What is happening here is some convention surrounding Rusts buffers (At least those that implement the <a href="https://doc.rust-lang.org/std/io/trait.Read.html#">Read</a> trait) that isn't exactly obvious. </p>
<p>If you go down the buffer rabbit hole far enough, you will find this rather vauge description on <code>std::io::Read</code>'s doc page:</p>
<blockquote>
<p>If the return value of this method is Ok(n), then implementations must guarantee that 0 <= n <= buf.len(). A nonzero n value indicates that the buffer buf has been filled in with n bytes of data from this source. If n is 0, then it can indicate one of two scenarios:</p>
<ol>
<li>This reader has reached its “end of file” and will likely no longer be able to produce bytes. Note that this does not mean that the reader will always no longer be able to produce bytes. As an example, on Linux, this method will call the recv syscall for a TcpStream, where returning zero indicates the connection was shut down correctly. While for File, it is possible to reach the end of file and get zero as result, but if more data is appended to the file, future calls to read will return more data.</li>
<li>The buffer specified was 0 bytes in length.</li>
</ol>
</blockquote>
<p>What this usually means in the context of a TCPStream, is that the other side closed the connection. When the TCPStream on the client side falls out of scope as the client unwinds, it
closes It's side. Since the buffer for the TCPStream will no longer produce any more bytes, it is considered closed. In <em>most</em> cases, you can interpret an <code>Ok(0)</code> result from this
method as a graceful disconnect. So lets modify the match block to reflect that.</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">if</span><span> ready.</span><span style="color:#96b5b4;">is_readable</span><span>() {
</span><span> buf = [</span><span style="color:#d08770;">0</span><span>; </span><span style="color:#d08770;">1024</span><span>];
</span><span> </span><span style="color:#b48ead;">match</span><span> stream.</span><span style="color:#96b5b4;">try_read</span><span>(&</span><span style="color:#b48ead;">mut</span><span> buf) {
</span><span> Ok(</span><span style="color:#d08770;">0</span><span>) => {
</span><span> println!("</span><span style="color:#a3be8c;">Client disconnected</span><span>");
</span><span> </span><span style="color:#b48ead;">return </span><span>Ok(());
</span><span> }
</span><span> Ok(n) => {
</span><span> println!("</span><span style="color:#a3be8c;">read </span><span style="color:#d08770;">{}</span><span style="color:#a3be8c;"> bytes</span><span>", n);
</span><span> </span><span style="color:#b48ead;">let mut</span><span> result_buffer: Vec<</span><span style="color:#b48ead;">u8</span><span>> = Vec::with_capacity(n);
</span><span> buf.</span><span style="color:#96b5b4;">take</span><span>(n as </span><span style="color:#b48ead;">u64</span><span>).</span><span style="color:#96b5b4;">read_to_end</span><span>(&</span><span style="color:#b48ead;">mut</span><span> result_buffer).await?;
</span><span> reply_queue.</span><span style="color:#96b5b4;">push</span><span>(result_buffer.</span><span style="color:#96b5b4;">to_vec</span><span>());
</span><span> }
</span><span> Err(</span><span style="color:#b48ead;">ref</span><span> e) </span><span style="color:#b48ead;">if</span><span> e.</span><span style="color:#96b5b4;">kind</span><span>() == io::ErrorKind::WouldBlock => {
</span><span> </span><span style="color:#b48ead;">continue</span><span>;
</span><span> }
</span><span> Err(e) => {
</span><span> </span><span style="color:#b48ead;">return </span><span>Err(e.</span><span style="color:#96b5b4;">into</span><span>());
</span><span> }
</span><span> }
</span><span> }
</span></code></pre>
<p>Now, when we get a 0 length buffer from the <code>try_read</code> method, the TCPStream handler method will exit, and all of its connections and buffers
will be automatically cleaned up as they fall out of scope.</p>
<p>Lets run the example code again:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span>Connection from </span><span style="color:#d08770;">127.0</span><span>.</span><span style="color:#d08770;">0.1</span><span>:</span><span style="color:#d08770;">58682
</span><span>read </span><span style="color:#d08770;">6</span><span> bytes
</span><span>Wrote </span><span style="color:#d08770;">6</span><span> bytes
</span><span>Client disconnected
</span></code></pre>
<p>While this behavior isn't very intuitive to people not already familiar, it can be dealt with in a pretty succinct way. And, as you can see, now that you know what it
means you have a program that will reliably handle all manner of connections, disconnections, reconnections, etc without having to address all those specific cases.</p>
<h1 id="some-other-pitfalls">Some Other Pitfalls</h1>
<h2 id="wait-how-did-i-just-send-data-to-a-disconnected-host">Wait, How Did I Just Send Data to a Disconnected Host?</h2>
<p>Well, the short answer is, you didn't. It just looked like you did.</p>
<p>Imagine this scenario:</p>
<p>Two systems have a long-running TCP Connection between them. </p>
<p>The server crashes or otherwise loses it's connection to the client.</p>
<p>The client then sends some data via the <code>try_write</code> method, which succeeds. What the heck?</p>
<p>Well, what we've succeeded in doing is writing data into a kernel networking buffer on a stream that, as far as the kernel is concerned, is still alive and healthy.</p>
<p>Networking problems are not uncommon, and many of them are recoverable. TCP is a protocol that, as mentioned in the intro, is capable of retrying and retransmitting lost data.
This takes time though, and by the time the complete retransmission process has finished, our program will have long moved on to another context (assuming it isn't blocking on
IO, which as previously established, is a bad thing.)</p>
<p>How has the kernel not realized there is a problem yet?</p>
<p>Well, in the case of Linux, the answer is hiding in proc:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>$ cat /proc/sys/net/ipv4/tcp_keepalive_time
</span><span>7200
</span><span>$ cat /proc/sys/net/ipv4/tcp_keepalive_probes
</span><span>9
</span><span>$ cat /proc/sys/net/ipv4/tcp_keepalive_intvl
</span><span>75
</span></code></pre>
<p>These are generally the default values for the three parameters of a TCP Connection that govern detecting failed connections.</p>
<p>As is so excellently explained by <a href="https://en.wikipedia.org/wiki/Keepalive">wikipedia</a>:</p>
<blockquote>
<ul>
<li>Keepalive time is the duration between two keepalive transmissions in idle condition. TCP keepalive period is required to be configurable and by default is set to no less than 2 hours.</li>
<li>Keepalive interval is the duration between two successive keepalive retransmissions, if acknowledgement to the previous keepalive transmission is not received.</li>
<li>Keepalive retry is the number of retransmissions to be carried out before declaring that remote end is not available</li>
</ul>
</blockquote>
<p>So, in the case of an idle connection, it will take more than 2 hours for a TCP connection to be considered failed, at which point calling <code>try_write</code> will result in an IOError.</p>
<p>But, there is something we can do about it. Upon creating TCP Connections, the Kernel will allow us to pass in our own values for each of these settings. Unfortunately, Tokio doesn't provide convenience methods for these as far as I can tell, so we need to do a little bit of excessive type conversions:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">let</span><span> stream = TcpStream::connect(&socket_addr_string).await.</span><span style="color:#96b5b4;">unwrap</span><span>()
</span><span style="color:#b48ead;">let</span><span> stream: std::net::TcpStream = stream.</span><span style="color:#96b5b4;">into_std</span><span>().</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span style="color:#b48ead;">let</span><span> socket: socket2::Socket = socket2::Socket::from(stream);
</span><span style="color:#b48ead;">let</span><span> keepalive = TcpKeepalive::new()
</span><span> .</span><span style="color:#96b5b4;">with_time</span><span>(Duration::from_secs(</span><span style="color:#d08770;">10</span><span>))
</span><span> .</span><span style="color:#96b5b4;">with_interval</span><span>(Duration::from_secs((</span><span style="color:#d08770;">5</span><span>));
</span><span>socket.</span><span style="color:#96b5b4;">set_tcp_keepalive</span><span>(&keepalive).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span><span style="color:#b48ead;">let</span><span> stream: std::net::TcpStream = socket.</span><span style="color:#96b5b4;">into</span><span>();
</span><span style="color:#b48ead;">let</span><span> stream: tokio::net::TcpStream = tokio::net::TcpStream::from_std(stream).</span><span style="color:#96b5b4;">unwrap</span><span>();
</span></code></pre>
<p>While TCP Keepalive is a pretty low-overhead operation, bear in mind that you probably don't want to set a one-second interval on a service
that maintains tens of thousands of connections.</p>
<p>Now, I do hear a few of you yelling at your monitor that I should have just constructed the socket using a <code>socket2</code> builder in the first place, but I disagree because of...</p>
<h2 id="dns-lookups-and-socket-address-type-conversions">DNS Lookups and Socket Address Type Conversions</h2>
<p>One thing you may want to keep in mind, especially if you are going to implement connection retry functionality into your program, is where your DNS lookups take place.
In a robust long-running daemon, you probably want to perform a DNS lookup on each successive connection, otherwise you may not discover until restarting your application
that the DNS records on the other side changed. This can result in really annoying behavior that frustrates operators.</p>
<p>So, where do DNS lookups take place?</p>
<p>A big clue can be gleaned by looking at the <a href="https://doc.rust-lang.org/std/net/enum.SocketAddr.html">SocketAddr</a> type:</p>
<pre data-lang="rust" style="background-color:#2b303b;color:#c0c5ce;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#b48ead;">pub enum </span><span>SocketAddr {
</span><span> </span><span style="color:#d08770;">V4</span><span>(SocketAddrV4),
</span><span> </span><span style="color:#d08770;">V6</span><span>(SocketAddrV6),
</span><span>}
</span></code></pre>
<p>Sure enough, there is no such thing as a <code>SocketAddr</code> that contains a DNS name, it must either be an IPv4 or IPv6 address + destination port. So, we can safely assume
that any time we convert from an address string to a <code>SocketAddr</code>, a DNS lookup must take place. This is one place where it is very easy for blocking IO to sneak it's
way into your otherwise async application. For instance, if you construct a socket in a non-async framework, and then convert it to an async socket, you may have performed
a non-async DNS lookup.</p>
<p>Now, in some cases you may want the lookup to only be performed once, in which case feel free to stick a conversion from string to <code>SocketAddr</code> directly in your <a href="https://docs.rs/clap/latest/clap/">clap</a> arguments, in which case your application will perform exactly one DNS lookup when the arguments are parsed and converted.</p>
<p>But, if that isn't the behavior you want, feel free to keep passing around you addresses as strings (or another intermediary type) for a bit longer.</p>
<h1 id="closing">Closing</h1>
<p>Don't be intimidated by lower level network protocols in Rust, or any language really. While they tend to require a little more work to get right than, say a HTTP REST call, they still have
their place in modern software engineering. Especially if you have a lot of data you need to get somewhere really fast with as little overhead as possible. </p>
<p>Just bear in mind, there are definitely some ways to shoot yourself in the foot. So make sure you test well, and have a basic understanding of what is actually going on
with the magical data portal that is a TCPStream.</p>
<p>You can access the complete source code used as an example <a href="https://github.com/graysonhead/rust-tcp-tutorial">here</a>.</p>
About2023-04-14T00:00:00+00:002023-04-14T00:00:00+00:00Unknownhttps://blog.graysonhead.net/about/<p>I'm Grayson. Here is some of the stuff I'm interested in:</p>
<ul>
<li>Software Engineering (Rust, Python, Nix, ...)</li>
<li>Compute Orchestration platforms (Openstack, Kubernetes)</li>
<li>Game Development</li>
<li>Offroad exploration</li>
<li>Car and boat restoration</li>
</ul>
<p>I'm also love aviation, and I'm currently working on getting my private license, so there will probably be some posts about that as well.</p>
<h2 id="contact">Contact</h2>
<p><a href="https://github.com/graysonhead">Github</a><br />
<a href="https://www.linkedin.com/in/graysonhead/">Linkedin</a><br />
<a href="https://hachyderm.io/@Darkside">Mastadon</a></p>
The Purpose of Grief2020-06-26T00:00:00+00:002020-06-26T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/greif/<img src="../../images/greif.jpg" alt="One of the last pictures I have of him and I together" />
<p>On a school morning, my sister and I sat with our mother eating breakfast, as we did almost every day. We lived in a rather eccentric but somewhat laid back trailer park mixed located effectively in the middle of a somewhat normal suburban neighborhood. We had our fair share of drama, drug dealers, and domestic calls. Sirens weren’t at all uncommon despite a relatively high level of safety. But the sirens this morning were different. One of my career mentors later in life, a former chaplain for the local fire department, would later tell me that nothing motivates emergency services quite like a call from a desperate mother saying the words “my child isn’t breathing.”</p>
<p>First there was one siren, then two, then what seemed like ten. Then there was Starflight, our local helicopter ambulance service. My mother offered a prayer for whomever was on the receiving end of all this attention, but otherwise we went on with our day. I remember not being able to focus in my classes, I remember wishing my best friend (and one of my only friends) was there. He had been sick for the last week with a stomach bug that was going around the school.</p>
<p>I couldn’t tell you what day it was, nor could I tell you whether school had just started or was about to end. I couldn’t tell you what classes I was in, or who my teachers were. But I can describe in detail when our teacher led us into the cafeteria with the rest of the 5th grade after our lass class. They had the folding tables laid out as if lunch were about to be served; with one notable exception. On each table, sat two tissue boxes. These had clearly been laid out with some care, they were perfectly centered on the tables and evenly spaced. I was confused as I walked in, and I was confused as I sat down, and I was confused when the principal started speaking. I don’t even remember what he said, I just remember the gist; your friend is dead. But I do remember sitting in that room on a bench surrounded by my peers, feeling like an open wound in a pool of alcohol. It was devastatingly impersonal. After all this was my best friend in all of the world; and I found out just like everyone else. The jealous anger only lasted a few seconds though as a new emotion was emerging. A feeling like perpetual falling. I would later learn the word for this feeling, Grief.</p>
<p>There are some things in life you just don’t get over, and Grief is one of those things. Grief is also a feeling impossible to describe, especially to someone who hasn’t experienced it. The dictionary only makes a half-baked attempt at defining it, labeling it as “deep sorrow”. Grief eviscerated me so severely that I thought I would never be happy again. In fairness, it did take several years. I cried most of the days I went to school, and unfortunately while my former peers were rather sympathetic to my tears (they themselves having gone through the same experience) I was no longer at that school, and a lot of my new classmates had no frame of reference for what we had gone through, and the bullies didn’t care. I didn’t have many friends, and I got into several fights. Most of the time I didn’t bother to defend myself, because I didn’t see the point. “Why bother? Everyone you love is mortal, they could all die at any moment. So what exactly is the point of making more friends?” My 11 year old self reasoned. “Why try with anything?”</p>
<p>Eventually my parents did something about my downward spiral, and sent me to a wonderful private school. It was a fresh start. Thanks to an exorbitant quantity of counseling, I could at least get through a normal school day without loosing it, and I stood a chance at making some friends; which I did. And there I met a very special girl, and even got enough courage to ask her out.</p>
<p>After I moved on to the next school I regressed somewhat. My childhood dog had died, and it was as if everything I fought so hard to repress burst out of me like a balloon. The scabs I had been building over the mental wounds of 4 years prior had been ripped off with impressive force. I got into fights again, my grades fell, and I dropped into depression again.</p>
<p>In the end, it turned out okay. I graduated, barely. I made some good friends, one of whom I still talk to and regard as one of my best friends. I helped make an online community that provided me with a good supply of long-distance friends that never made me feel judged or unwelcome. And, best of all, the aforementioned girl reconsidered my offer a few years later. We’ve been together for almost 11 years now. Together, we created a life that I love.</p>
<p>I suffered more Grief after that. Some of my online friends were killed by a drunk driver. My grandmother died. And more recently, my wife and I experienced a miscarriage. It was never the same twice. And for certain, all of those paled in comparison to that first brush with Grief. But they still sent me dropping into the slick ditch that seems impossible to climb out of.</p>
<p>I became desperate to understand what the purpose of Grief was in my life.</p>
<p>I read on the internet somewhere that Grief is like waves. The first ones are 100ft tall and 10 seconds apart, but eventually they come farther and farther apart. They still sweep you off of your feet, but you can see the coming. You know what the triggers of Grief are. And then they get shorter, instead of being 100 feet, they might be 20 feet. I started thinking about how tall the waves were and at what frequency they occurred during my life. In the case of my friend, they were 100 feet high and came at least once or twice a week for the first two years. After that, they came around twice a month. I think the first time one of the waves hit me and failed to bring any tears was my Junior year of high school.</p>
<p>The other losses in my life have followed the same pattern, but they usually had shorter waves to begin with, and the frequency was much lower. As if I was starting in year three after the initial event. I thought that further losses would get easier, but that didn’t seem to be the case. Each pain had its own distinct flavor, and strangely the waves didn’t seem to ever coincide. Some were above average, and some less so. But they all followed the same pattern. A wave, the normalcy, then another wave.</p>
<p>My son was born on July 17th. The pregnancy was normal, the birth was normal, and despite a small scare at the start (he was to stubborn to cry, which concerned the nurse) he had no medical issues to speak of if you overlooked his bruised and misshapen head (even the perfect birth is still traumatic.) As the nurses tended to my wife, I sat with him on the rather spartan couch that had caused me so much ire over the previous 24 hours (a block of concrete would have been more comfortable). I looked at his face, and his eyes were looking deep into mine. I was overwhelmed by a sense of Joy that was indescribable. That joy never left me. I still feel it to this day. My family makes me so immensely happy that my sides often feel like they might burst.</p>
<p>A few months later, I looked at my son as I was feeding him a bottle, and I saw my friend looking back at me. At first, I was confused. Then I was scared as the realization that I brought my son into a dangerous and unpredictable world hit me. But I was also strangely comforted. It was a mixture of melancholy and contentedness. At this moment, I realized what the purpose of Grief was in my life.</p>
<p>You can’t truly appreciate joy until you’ve experienced despair.</p>
<p>Grief is the black in a monochrome image. Without the contrast, you don’t properly appreciate the beauty. The more sadness and darkness in your life, the more that joy and happiness are brought into relief. It takes knowing the acidic taste of a lemon to appreciate the sweetness of a strawberry.</p>
<p>If I have anything to thank those who have departed from my life for, its that I don’t take much for granted. Often times I’ll be tired after a long day at work, and my toddler will be getting into something he isn’t supposed to for the 6th or 7th time; and as I’m on the verge of loosing my temper a little wave of Grief will hit me, and I’ll take a moment to appreciate what I have in my life.</p>
<p>When I first felt the waves of Grief after my friend died, I never thought the waves would stop destroying me. I never thought I was going to be okay ever again.</p>
<p>The waves never did stop coming.</p>
<p>But, Nearly 15 years later, I feel like I’m standing ankle deep in the water of a lake. Occasionally, small waves no more than a few inches high will come up to the shore in batches of three or four. They are big enough that you notice them, but they cause no discomfort. They remind me of the 100ft tall tsunamis that destroyed me a decade ago, but they are also comforting. They reassure me that I wont forget the people who have departed from my life. They remind me of all the good memories I’ve had with them. And they remind me not to take the simplest things for granted. At this point, I don’t mind the waves. They bring me a sort of melancholic comfort. There will still be a big wave every now and then, but the water is calm enough that I’ll see it coming. I think the comfort comes from the fact that I know I have loved as deep as I possibly can, and tried to make the most of every singe day. When my time comes, I know I won’t have regrets. And, at least partially, I have Grief to thank for that.</p>
<p>If you are currently suffering from loss, just know one thing; It does get better. It takes a long time. But it does get better.</p>
Your Managed Service Provider Sucks2019-10-24T00:00:00+00:002019-10-24T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/your-msp-sucks/<p>One of the best jobs I ever had was at a small local managed service provider and software development firm. The constant customer interaction, plethora of technical challenges with insufficient manpower to fix them, and the companies desire to actually make money turned my coworkers and I into some of the most well-rounded IT engineers I’ve ever met. Yes, we can design you a public cloud on Vmware, Cloudstack, or Openstack. Sure, we can set up BGP edge routing for you. You need some basic programming done to integrate two products with sane APIs? Sure thing. And somehow we still found room to store gems such as “How to Read a P&L”, “Contract Writing 101”, and “How to Convince People to Sign 6 Figure Checks”.</p>
<p>Before you think that I’m writing this just to gasconade, I was far from the only employee that had a story like this. The secret sauce in our ability to step into an unfamiliar situation and get shit done, was that we were constantly in an unfamiliar situation. And we lived in an environment that had us near constantly in an unfamiliar or uncomfortable situation. We were the people that showed up when the single IT guy for a medium sized company was out of ideas, and we were the people that showed up to a new customer without backups that just lost data that essentially comprised their company, and we were the people that showed up after one of your IT people went rouge and tried to do as much damage as possible on the way out. Our world was filled with constant stress and worry that this might be the time our ability to read a manual and keep a cool head would finally fail us. Occasionally it did.</p>
<p>At the time, I hated it. There was no stability, no rest, and an endless torrent of other peoples problems that I had never seen before. But if I could go back and change anything about my career, I wouldn’t change my time at MSPs (but I also wouldn’t go back). It helped that I had a pair of amazing mentors, one of whom was a seasoned Systems Engineer/Programmer, and the other one happened to own the company. It was the perfect launchpad for my college-free career path, it was like rocket fuel for my career.</p>
<p>If you have the opportunity to work at an MSP, you should. Its experience that money can’t buy. But this article isn’t targeted at people who are looking for jobs, this article is targeted at people who are either currently running, or thinking about running MSPs.</p>
<p>All told now, I’ve lived at MSPs for a little less than a decade. I’ve worked for the big and small, old and new, great and horrible. I worked at one for 6 + years, and I’ve also worked for 3 at the same time. I’ve been everything from Tier 1 help-desk to manager to director. All told, if you count little stints of contracting here and there, I’ve either contracted or worked for about 8 different MSPs.</p>
<p>Without early and intentional intervention, even a successful MSP will have a tendency to fall into the “traps” I’m about to outline. If you are thinking about starting an MSP, pay close attention. I’m about to give you a glimpse into your future, and believe me, it sucks.</p>
<h2 id="you-aren-t-charging-enough-but-you-can-t-charge-enough">You Aren't Charging Enough, But You Can't Charge Enough</h2>
<p>Any kind of outsourcing is a race to the bottom. Standard rates for normal per-head tech support now is around $45–70 per person per month. This is less than most people’s cell phone bills. And granted, I’m talking remote-only basic computer support, but this is the bread and butter of most MSPs. When each of your customer’s employees is demanding a half an hour of your time on average each month (and this is probably a little low for the typical customer that would hire an MSP in the first place), you are talking about dollars of profit per head after the employee helping the customer gets paid. So the only thing you can do is start adding on services. A few years ago, this would actually get you a pretty decent living. Most tech companies had a serious bottleneck regarding marketing and sales, and they would give you 20–30% margins just to introduce their email platform, antivirus software, or productivity suite to your customers. And for hardware sales, the margins were often bigger. I remember selling a customer 50 computers at a 47% margin as a result of a few concurrent promotions our distributor was doing. That damn near covered the operating cost for that MSP for 2 months. But nowadays? You are lucky if you see 5%. And its hardly free money, you have to work at it. The amount of “mailbox money” you are getting from any modern SaaS company that you sell into your customer base is probably going to offset the cost of postage, but not much more.</p>
<p>So unless you are able to push datacenter gear in large quantities, where 20–25% margins are still realistic, this isn’t going to net you any profits. At best, it might help you close the gap between MRR and operational costs, but in all honesty its probably going to be more of a distraction than anything.</p>
<h2 id="you-need-to-use-the-same-contract-for-all-of-your-customers">You Need to Use the Same Contract for ALL of Your Customers</h2>
<p>This is by far one of the most important things you can do to offset the MSP pain. If one of your customers doesn’t want to get on board, drop them.</p>
<p>You can design a contract that is just flexible enough, while also allowing you to standardize how you get paid, and how you provide services for your customers. My recommendation would be to create a document that outlines all of your services, and then have a separate form specific to each customer that lays out which of those services they have purchased for you. Its super easy. You are generally going to wind up with a 3 tier “basic — essential — premium” sort of model. And there is going to be a standard distribution of customers across these three tiers, with most picking essential and a few big fish picking premium. And those big fish are where you are actually going to make some money. But don’t worry, you are going to spend all that money you just made trying to acquire more customers like them.</p>
<p>The most important reason for this, is consistency. Your employees need to understand how you make money, and they need to understand what they are and aren’t allowed to do for the customer without having to get a check. You can’t expect them to remember the special snowflake contract for each of your customers, and you can’t expect them to look it up every time Sally from Accounting at that underfunded non-profit needs help changing her toner cartridge. Any time they spend doing overhead tasks is time they aren’t spending making your customers happy.</p>
<h2 id="if-your-operational-costs-aren-t-covered-by-your-recurring-revenue-your-time-is-limited">If Your Operational Costs Aren’t Covered by Your Recurring Revenue, Your Time Is Limited</h2>
<p>This one is pretty simple. If you can’t find a way to cover your operational costs with your recurring revenue, you are going to struggle and flail every month selling more labor than you have to try and cover the gap. That works for a few months, but eventually you accumulate more promises than you have people to fulfill them, and then you are screwed with no way out. Don’t do this. Cover your MRR from the start, when your operational costs are small, and don’t expand till you have MRR to cover the expansion.</p>
<p>And for heaven’s sake, make your MRR customers sign contracts. The term doesn’t have to be ridiculously long, but you need time to find replacement customers if one of your big fish decides to leave.</p>
<h1 id="a-better-model">A Better Model</h1>
<h2 id="write-an-operations-manual">Write an Operations Manual</h2>
<p>Unless you hate taking vacations.</p>
<p>If you write it good enough, and you hire the right people, your training will consist of you handing it to people. For everyone else, it serves as an unambiguous reference in situations where you would normally get a phone call at a funeral and be “that uncle”.</p>
<h2 id="your-first-employee-ideally-you-should-be-a-programmer">Your First Employee (Ideally You) Should Be a Programmer</h2>
<p>MSPs are tech companies, and MSPs require high operational efficiency. The kind of operational efficiency you aren’t going to get by buying off the shelf generic software. By the time you get around to hiring your first few Technicians, you should damn well know exactly what your workflow looks like. And you should then go and create workflow automation tools that fit that workflow like a glove. Then just keep building. Make a customer portal (it makes you look way more professional than your competitors with generically branded Connectwise portals), and then integrate it with everything you sell. A customer should be able to log into a website and see all existing tickets, incidents, upcoming bills, and all of the services they can buy from you. Make it easy, and make it completely unambiguous. Ambiguity is what kills operational efficiency, and workflow automation software kills ambiguity.</p>
<p>But this is a 10,000ft view. You could write a book on what I’ve said above. All I’ll say is; if you don’t know how to do it, find someone that does. But I’ll also say that if you have to find someone that knows how to do it, you should probably consider a different career unless you have a large pile of money you want to light on fire.</p>
<h2 id="revolving-door-hiring">Revolving Door Hiring</h2>
<p>This is the most difficult thing, and the most worthwhile thing. You need to build your company in a way that it is accepting of entry level technicians, and give them just enough guard rails to help them get started, but not enough that they feel restricted. Find people that have built their own computers, find people that have modded video games, find people that use Linux (even if you are a windows shop.) Hire exclusively on three factors; Are they self-driven? Do they like learning new skills? Do they dress and speak presentably?</p>
<p>Eventually you are going to reach a point where your employees out-learn the position they occupy. Due to the nature of MSPs, you probably won’t have grown enough to offer your first few senior employees the positions that are worthy of their new skill-sets. What you do at this moment is critical, many people will stretch and find money to promote their first employees to positions that don’t exist, and that the business can’t justify. Don’t do it. Give them the most glowing letter of recommendation ever, and don’t fire them before they find a new job, but you need to let the door revolve. MSPs serve an important role in the tech job industry, they are the shitty jobs everyone takes as an entry level tech to go find a less shitty job once they have some experience. If you are just starting out, its unlikely that you are going to be able to pay to retain top tier talent; so don’t waste your money or time trying. Just let the door revolve, and go start a few more careers.</p>
<h2 id="find-out-what-you-are-really-good-at-and-stop-doing-everything-else">Find Out What You Are Really Good At, and Stop Doing Everything Else</h2>
<p>If you are successful, hire the right people, and automate your workflows to a militant degree, you will eventually find a niche.</p>
<p>One of the companies I worked for found a niche in Business Continuity services. Our mistake was not pivoting to doing BCS exclusively while we had the opportunity to do so. If you look at a lot of software companies in the MSP space, they found a niche and then started selling their software or services to other MSPs. This is the dream. Even if you stay in the service industry, its way easier to sell yourself as a specialized provider (or better yet, software provider that offers services) than generic catch all tech support. And there is the possibility you will actually make money. So when you see your opportunity to jump ship and fill a niche; Do it.</p>
<p>All MSP markets outside of dedicated government contractors are a race to the bottom. You might not cut costs, but your competition will. And they are probably better at it than you.</p>
Runcible: Declarative Network Automation2019-05-12T00:00:00+00:002019-05-12T00:00:00+00:00Unknownhttps://blog.graysonhead.net/posts/runcible/<p>I’m probably better with systems than networks. I’ve got a decent grasp of Networking concepts, but I’ve not worked in large scale environments as a network engineer. I architect-ed the network for a few small (<50 rack) datacenters that were adequate, played with a lot of Linux powered switches, and did a fair bit of edge design with BGP.</p>
<p>On the other hand, as a systems guy I’ve worked in two notable large scale environments at this point, both of which are in the top 500 website list globally according to Alexa (the website statistics service, not the lady-tube). As well as countless smaller, but still pretty large environments. In these environments, some kind of config management for physical infrastructure is absolutely not optional; and can be the difference between being able to scale or being crushed under your own weight. As a result, I’ve developed some strong opinions about configuration management from a user’s perspective.</p>
<p>A while back, I found myself in the position to manage a network again after doing large-scale systems work for a bit, and came to a realization: Open source declarative network automation tools don’t exist.</p>
<p>Now, I know you can use Ansible, Chef, Puppet, and Salt to do a lot of network automation. And I’ve used Ansible and Salt to manage a mixed Juniper/Cumulus environment on several occasions. But trust me when I say the tools aren’t designed to manage networks, they were adapted to it. And if you are a network engineer that hasn’t spent any time on the other side of the fence, you probably don’t know what you are missing out on.</p>
<h1 id="the-first-problem-commonality-a-dependency-for-dry-declarations">The First Problem: Commonality (A Dependency for Dry Declarations)</h1>
<p>The rule of DRY (Don’t Repeat Yourself) exists for your sanity, and also the longevity of your keyboard. Every programmer lives and dies by DRY. And good configuration management languages should embrace it as well.</p>
<p>But with existing tools, you don’t really get it. At least not for network hardware.</p>
<p>Almost all of the major config automation suites support some kind of hierarchical hash/dict deep merge. For those unfamiliar with the concept, it goes something like this.</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#bf616a;">layers</span><span>:
</span><span> - </span><span style="color:#a3be8c;">country
</span><span> - </span><span style="color:#bf616a;">statecountry</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">USA
</span><span> </span><span style="color:#bf616a;">national_bird</span><span>: </span><span style="color:#a3be8c;">Bald Eagle
</span><span> </span><span style="color:#bf616a;">national_mammal</span><span>: </span><span style="color:#a3be8c;">Bison
</span><span> </span><span style="color:#bf616a;">national_flower</span><span>: </span><span style="color:#a3be8c;">Rose
</span><span> </span><span style="color:#bf616a;">national_tree</span><span>: </span><span style="color:#bf616a;">Oak Treestate</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Texas
</span><span> </span><span style="color:#bf616a;">state_flower</span><span>: </span><span style="color:#a3be8c;">BlueBonnet
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Utah
</span><span> </span><span style="color:#bf616a;">state_flower</span><span>: </span><span style="color:#a3be8c;">Sego Lilydeep_merge(country.USA, state.Texas)
</span><span style="color:#bf616a;">national_bird</span><span>: </span><span style="color:#a3be8c;">Bald Eagle
</span><span style="color:#bf616a;">national_mammal</span><span>: </span><span style="color:#a3be8c;">Bison
</span><span style="color:#bf616a;">national_flower</span><span>: </span><span style="color:#a3be8c;">Rose
</span><span style="color:#bf616a;">national_tree</span><span>: </span><span style="color:#a3be8c;">Oak Tree
</span><span style="color:#bf616a;">state_flower</span><span>: </span><span style="color:#a3be8c;">BlueBonnetdeep_merge(country.USA, state.Utah)
</span><span style="color:#bf616a;">national_bird</span><span>: </span><span style="color:#a3be8c;">Bald Eagle
</span><span style="color:#bf616a;">national_mammal</span><span>: </span><span style="color:#a3be8c;">Bison
</span><span style="color:#bf616a;">national_flower</span><span>: </span><span style="color:#a3be8c;">Rose
</span><span style="color:#bf616a;">national_tree</span><span>: </span><span style="color:#a3be8c;">Oak Tree
</span><span style="color:#bf616a;">state_flower</span><span>: </span><span style="color:#a3be8c;">Sego Lily
</span></code></pre>
<p>If you didn’t have deep merge available, you would have to define each entry like this:</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#bf616a;">item1</span><span>:
</span><span> </span><span style="color:#bf616a;">state_name</span><span>: </span><span style="color:#a3be8c;">Texas
</span><span> </span><span style="color:#bf616a;">national_bird</span><span>: </span><span style="color:#a3be8c;">Bald Eagle
</span><span> </span><span style="color:#bf616a;">national_mammal</span><span>: </span><span style="color:#a3be8c;">Bison
</span><span> </span><span style="color:#bf616a;">national_flower</span><span>: </span><span style="color:#a3be8c;">Rose
</span><span> </span><span style="color:#bf616a;">national_tree</span><span>: </span><span style="color:#a3be8c;">Oak Tree
</span><span> </span><span style="color:#bf616a;">state_flower</span><span>: </span><span style="color:#a3be8c;">BlueBonnet
</span><span> </span><span style="color:#bf616a;">country_name</span><span>: </span><span style="color:#bf616a;">USAitem2</span><span>:
</span><span> </span><span style="color:#bf616a;">state_name</span><span>: </span><span style="color:#a3be8c;">Utah
</span><span> </span><span style="color:#bf616a;">national_bird</span><span>: </span><span style="color:#a3be8c;">Bald Eagle
</span><span> </span><span style="color:#bf616a;">national_mammal</span><span>: </span><span style="color:#a3be8c;">Bison
</span><span> </span><span style="color:#bf616a;">national_flower</span><span>: </span><span style="color:#a3be8c;">Rose
</span><span> </span><span style="color:#bf616a;">national_tree</span><span>: </span><span style="color:#a3be8c;">Oak Tree
</span><span> </span><span style="color:#bf616a;">state_flower</span><span>: </span><span style="color:#a3be8c;">Sego Lily
</span><span> </span><span style="color:#bf616a;">country_name</span><span>: </span><span style="color:#a3be8c;">USA
</span></code></pre>
<p>We saved 2 lines of defining by having an inheritance structure, and you can probably tell that the more attributes we define at the country level, the more savings we have.</p>
<p>Merging can save you a lot of pain and boilerplate, especially when you have different servers in different locations with different configurations, but they still share things in common (like what they do, and how the software gets installed.) For servers, the OS serves as an adequate abstraction layer to smooth out hardware differences.</p>
<p>The problem is, no one has the same kind of network hardware at the end of the day, and in a lot of cases you may not even be able to run the same kind of network operating system in two different datacenters, or often, in the same datacenter. If you have an extremely homogeneous environment, you might not notice this problem, but chances are better that you don’t. And even if you do now, you won’t in the future when you inevitably switch platforms.</p>
<p>Using Ansible as an example, lets look at two different configuration snippets:</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#bf616a;">Juniper:- name</span><span>: </span><span style="color:#a3be8c;">Configure interface in access mode
</span><span> </span><span style="color:#bf616a;">junos_l2_interface</span><span>:
</span><span> </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">ge-0/0/1
</span><span> </span><span style="color:#bf616a;">description</span><span>: </span><span style="color:#a3be8c;">interface-access
</span><span> </span><span style="color:#bf616a;">mode</span><span>: </span><span style="color:#a3be8c;">access
</span><span> </span><span style="color:#bf616a;">access_vlan</span><span>: </span><span style="color:#d08770;">2
</span><span> </span><span style="color:#bf616a;">active</span><span>: </span><span style="color:#d08770;">True
</span><span> </span><span style="color:#bf616a;">state</span><span>: </span><span style="color:#bf616a;">presentCumulus:- name</span><span>: </span><span style="color:#a3be8c;">Add an access interface on swp1
</span><span> </span><span style="color:#bf616a;">nclu</span><span>:
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">add int swp1
</span><span> - </span><span style="color:#a3be8c;">add int swp1 bridge access 2
</span></code></pre>
<p>Both of these configurations do the exact same thing, but as you can see, there is no commonality between them whatsoever. As a result, you can’t have an inheritance structure, which means any changes made to one environment have to be translated by a human and inserted into the other(s). This completely breaks the principle of inheritance.</p>
<p>Moreover, there is no commonality between different types of network hardware (with Juniper being a partial exception to this). Your Cisco Routers and Palo Alto firewalls don’t benefit from the extensive and labeled VLAN dictionary that you use to drive your switch configurations, so you have to repeat information even more.</p>
<h1 id="the-second-problem-topology-is-a-thing">The Second Problem: Topology is a Thing</h1>
<p>Topology isn’t a very important concept in server automation. Things need to be grouped, and you definitely want to keep a few of your application nodes untouched as you upgrade the others. In a lot of cases, you can define a sort of dependency hierarchy, but this doesn’t quite cut it for a network. Networks are all about topology, and trying to pretend like it isn’t important when running automation is a path to ruin.</p>
<p>Take the following simple topology:</p>
<img src="../images/runcible1.png" alt="Image of a switch topography depicting an expanded spine with 6 spine switches and 8 leaf switches. The spine swithces are labeled as group 1, and the leaf switches group 2" />
<p>Group 1 is the role <code>core_switches</code>, and Group 2 is the role <code>rack_switches</code>.</p>
<p>All configuration management suites will give you the option to do a rolling upgrade within a group, which has a high probability of not working in this example. If you are running your operations idempotently, it may not cause a problem, but if you are making a data-center-wide change to your network environment that requires a restart of interfaces; the random choices your automation will make in this case could shoot you in the foot.</p>
<p>If you allow it to run two concurrent applies, and it picks the two middlemost core switches, you’ve just split-brained your datacenter. Or, slightly less worse, it might do a simultaneous apply on both of the switches in a rack, taking the rack down.</p>
<p>The current way to fix this? Just only run against one device within a role at any given time. The problem with this? Some datacenters have a LOT of network devices, as there are only so many hours in the day…</p>
<p>Good network automation tools need to be aware of concepts like adjacency, clustering link aggregation, and pathing. Otherwise you are stuck automating things at a snails pace, or performing dangerous actions.</p>
<h1 id="the-third-problem-bootstrapping">The Third Problem: Bootstrapping</h1>
<p>Lets assume you have a completely homogeneous environment, and you have managed to solve the topology problem mentioned above. If you wind up having to deploy 30 racks, what does your deployment workflow look like? For servers, there is generally a middleman (generally PXE based) that deals with the initial provisioning of a system, and getting it “up enough” into a state where provisioning can be taken over by another system. But many switch ecosystems don’t have this.</p>
<p>What usually happens is that a network engineer will connect a serial cable to a switch, and manually configure it’s management interface to be reachable over SSH or another protocol, then your config management software takes over. More rarely, your ecosystem will support some kind of “phone home” DNS address that can be used to phone home to a server and fetch a configuration file. But for consultants working in smaller environments, this is a non-starter, as that infrastructure doesn’t exist yet (that is what they are in the process of creating.) Also, that infrastructure can’t exist until the network does, so it makes little sense for provisioning network devices.</p>
<p>All of this combines to make the process of bootstrapping switches specifically (but also a horde of other network devices) a royal pain, and often a more annoying task than provisioning a far greater number of servers.</p>
<h1 id="what-is-runcible">What is Runcible?</h1>
<p>Runcible is a Framework and Command Line Application written in python to provide a declarative abstraction layer for configuring network devices. Put simply, it takes a desired state of a set of network appliances (that the user specifies), compares it to the current state of those appliances, and then determines all of the commands that need to be run to correct the differences idempotently. Due to it’s preference for operating on the command-line by default (you can also write plugins that interact with REST APIs), you can utilize the same plugins to configure devices over SSH, Telnet, RS-232, or any text-based CLI.</p>
<h1 id="how-does-it-fix-those-problems">How Does it Fix Those Problems?</h1>
<h2 id="modular-interfaces">Modular Interfaces</h2>
<p>Network devices generally need to comply with networking standards, and Runcible takes advantage of the resultant similarities and provides abstraction layers that can be re-used on multiple different platforms with minimal modification.</p>
<p>As an example, here is a Runcible switch definition for a Cumulus switch in YAML:</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#bf616a;">meta</span><span>: </span><span style="color:#65737e;"># <- Meta module
</span><span> </span><span style="color:#bf616a;">device</span><span>:
</span><span> </span><span style="color:#bf616a;">ssh</span><span>:
</span><span> </span><span style="color:#bf616a;">hostname</span><span>: </span><span style="color:#a3be8c;">switch1.oob.domain.net
</span><span> </span><span style="color:#bf616a;">username</span><span>: </span><span style="color:#a3be8c;">cumulus
</span><span> </span><span style="color:#bf616a;">default_management_protocol</span><span>: </span><span style="color:#a3be8c;">ssh
</span><span> </span><span style="color:#bf616a;">driver</span><span>: </span><span style="color:#a3be8c;">cumulus
</span><span style="color:#bf616a;">system</span><span>: </span><span style="color:#65737e;"># <- Module
</span><span> </span><span style="color:#bf616a;">hostname</span><span>: </span><span style="color:#a3be8c;">switch1.inf.domain.net </span><span style="color:#65737e;"># <- Attribute
</span><span style="color:#bf616a;">interfaces</span><span>: </span><span style="color:#65737e;"># <- Module
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">swp1 </span><span style="color:#65737e;"># <- Sub-module /attribute
</span><span> </span><span style="color:#bf616a;">pvid</span><span>: </span><span style="color:#d08770;">22
</span><span> </span><span style="color:#bf616a;">bpduguard</span><span>: </span><span style="color:#d08770;">False
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">swp2
</span><span> </span><span style="color:#bf616a;">pvid</span><span>: </span><span style="color:#d08770;">23
</span><span> </span><span style="color:#bf616a;">bpduguard</span><span>: </span><span style="color:#d08770;">True
</span><span> </span><span style="color:#bf616a;">portfast</span><span>: </span><span style="color:#d08770;">True
</span></code></pre>
<p>This example defines three modules, meta, system, and interfaces. Meta is a special module that contains configuration information, and also selects the driver; which is cumulus in this case. The other modules provide an interface to declare the desired state of certain aspects of the switch.</p>
<p>In the system module, we are defining a single attribute; hostname. This controls the hostname of the switch. In the interfaces module, we are defining two sub-modules of the interface type, swp1 and swp2. Inside of these sub-modules, attributes are defined that define the behavior for the individual switchports on the system.</p>
<p>The advantage to these interfaces, is that they are as generic as possible. They try to conform to the standards that each network device follows, not to the language and naming schemes of the individual network device platforms themselves. Each plugin and provider (we will get to those later) can provide their own modules; but they should consume the default ones when possible. This allows the same definition to be applicable to multiple types of network devices of a similar class. In theory, if a switch from “vendor A” supports the same modules as a switch from “vendor B”, their definition files can be completely interchangeable.</p>
<p>Lets look at a practical example of this:</p>
<pre data-lang="python" style="background-color:#2b303b;color:#c0c5ce;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b48ead;">from </span><span>runcible.api </span><span style="color:#b48ead;">import </span><span>Device
</span><span>
</span><span>vlans = {
</span><span> "</span><span style="color:#a3be8c;">vlans</span><span>": [
</span><span> {"</span><span style="color:#a3be8c;">name</span><span>": "</span><span style="color:#a3be8c;">Employee Vlan</span><span>", "</span><span style="color:#a3be8c;">id</span><span>": </span><span style="color:#d08770;">20</span><span>},
</span><span> {"</span><span style="color:#a3be8c;">name</span><span>": "</span><span style="color:#a3be8c;">Super Secret Secure Vlan</span><span>", "</span><span style="color:#a3be8c;">id</span><span>": </span><span style="color:#d08770;">4000</span><span>}
</span><span> ]
</span><span>}
</span><span>
</span><span>interface_template = {
</span><span> "</span><span style="color:#a3be8c;">interfaces</span><span>": [
</span><span> {"</span><span style="color:#a3be8c;">name</span><span>": "</span><span style="color:#a3be8c;">swp1</span><span>", "</span><span style="color:#a3be8c;">pvid</span><span>": </span><span style="color:#d08770;">22</span><span>, "</span><span style="color:#a3be8c;">bpduguard</span><span>": </span><span style="color:#d08770;">False</span><span>},
</span><span> {"</span><span style="color:#a3be8c;">name</span><span>": "</span><span style="color:#a3be8c;">swp2</span><span>", "</span><span style="color:#a3be8c;">pvid</span><span>": </span><span style="color:#d08770;">23</span><span>, "</span><span style="color:#a3be8c;">bpduguard</span><span>": </span><span style="color:#d08770;">True</span><span>}
</span><span> ]
</span><span>}
</span><span>
</span><span>switch_1 = {
</span><span> "</span><span style="color:#a3be8c;">meta</span><span>": {
</span><span> "</span><span style="color:#a3be8c;">device</span><span>": {
</span><span> "</span><span style="color:#a3be8c;">ssh</span><span>":{
</span><span> "</span><span style="color:#a3be8c;">hostname</span><span>": "</span><span style="color:#a3be8c;">192.168.1.1</span><span>",
</span><span> }
</span><span> },
</span><span> "</span><span style="color:#a3be8c;">default_management_protocol</span><span>": "</span><span style="color:#a3be8c;">ssh</span><span>",
</span><span> "</span><span style="color:#a3be8c;">driver</span><span>": "</span><span style="color:#a3be8c;">vendor-a</span><span>"
</span><span> },
</span><span> "</span><span style="color:#a3be8c;">system</span><span>": {
</span><span> "</span><span style="color:#a3be8c;">hostname</span><span>": "</span><span style="color:#a3be8c;">switch1</span><span>"
</span><span> }
</span><span>}
</span><span>
</span><span>switch_2 = {
</span><span> "</span><span style="color:#a3be8c;">meta</span><span>": {
</span><span> "</span><span style="color:#a3be8c;">device</span><span>": {
</span><span> "</span><span style="color:#a3be8c;">ssh</span><span>":{
</span><span> "</span><span style="color:#a3be8c;">hostname</span><span>": "</span><span style="color:#a3be8c;">192.168.1.2</span><span>",
</span><span> }
</span><span> },
</span><span> "</span><span style="color:#a3be8c;">default_management_protocol</span><span>": "</span><span style="color:#a3be8c;">ssh</span><span>",
</span><span> "</span><span style="color:#a3be8c;">driver</span><span>": "</span><span style="color:#a3be8c;">vendor-b</span><span>"
</span><span> },
</span><span> "</span><span style="color:#a3be8c;">system</span><span>": {
</span><span> "</span><span style="color:#a3be8c;">hostname</span><span>": "</span><span style="color:#a3be8c;">switch2</span><span>"
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#b48ead;">for </span><span>configuration </span><span style="color:#b48ead;">in </span><span>[switch_1, switch_2]:
</span><span> configuration.</span><span style="color:#bf616a;">update</span><span>(vlans)
</span><span> configuration.</span><span style="color:#bf616a;">update</span><span>(interface_template)
</span><span> device = </span><span style="color:#bf616a;">Device</span><span>(configuration["</span><span style="color:#a3be8c;">system</span><span>"]["</span><span style="color:#a3be8c;">hostname</span><span>"], configuration)
</span><span> device.</span><span style="color:#bf616a;">plan</span><span>()
</span><span> device.</span><span style="color:#bf616a;">execute</span><span>()
</span></code></pre>
<p>As you can see, we have two different devices with two different drivers that support the system, interfaces, and vlans module. Assuming you have a valid plugin for both “vendor-a” and “vendor-b”, you will get effectively the same configuration on both switches (aside from the hostname in this example.)</p>
<p>Intuitively, adding another vlan to the “vlans” list will add it to both of the switches that merge that part of the configuration.</p>
<p>If you aren’t a Python developer, don’t worry. Later versions of Runcible with a more complete CLI component will abstract this behavior into a JSON directory structure. No code required.</p>
<h2 id="topology-aware-schedulers">Topology Aware Schedulers</h2>
<p>One of the priorities of Runcible in regards to topology discovery, is flexibility. Many devices support discovery protocols such as LLDP and CDP, some vendors offer proprietary methods of discovery, many users will want to define the topology manually, and some will be fine with grouping. A lot of people (especially consultants working in a large number of smaller networks) are probably fine with a naive scheduler that runs on one device at a time until they are all done.</p>
<p>As for the decisions that are informed by the topology, that remains to be seen somewhat. Again, the focus will be on flexibility. For many changes, it would be better to have a core -> out run order, for others an edge -> core run order. For less fancy operations, it would probably be fine to determine the run order by choosing devices with the least adjacent runs, or just ensure that only one cluster member at a time is being operated on for highly available devices.</p>
<p>But this will also be one of the final steps in a 1.0 release. The focus for now is a bottom to top implementation of the API functions, meaning that the device API will be more or less complete before the scheduler API is ready, and then a CLI application to wrap all of those will follow.</p>
<h2 id="operating-on-the-cli-or-whatever-really">Operating on the CLI (Or Whatever, Really)</h2>
<p>By default, Runcible operates on devices using CLI commands. While this might seem a bit backwards in the {currentyear} REST JSONAPI fervor, it actually has a lot of wonderful side effects. Firstly, any text-based terminal will suffice for configuration. This means that when you go to the datacenter to deploy some switches, you just bring your laptop and clone your definition files locally. Then just plug your serial cable into a switch, and hit go.</p>
<p>Its a simple concept, but very powerful.</p>
<h1 id="so-when-can-i-get-it">So, When Can I Get It?</h1>
<p>Right now, this is a project I’m working on in my spare time, so it might be a while. But I’ve been working on solving this problem for a while now, and I don’t have any intention of stopping. I also welcome contributors, so if you want to bring this about faster; be the change you seek in the world!</p>
<h2 id="i-want-to-help">I Want to Help!</h2>
<p>Great! I look forward to working with you!</p>
<p>The best place to get started is here: https://runcible.readthedocs.io/en/latest/contribution_guide.html</p>
<p>Right now you won’t find much, and I haven’t posted any issues on GitHub yet, as the internal class structure is still somewhat in flux. Over the few months I’m hoping to get two somewhat feature complete plugins written for Cumulus and Unifi switches. Once these are complete I will begin a big documentation push in an effort to make the project more friendly to new contributors. Around then, I’ll make an announcement that it is “safe” to make pull requests (when the Plugin structure is stable); but until then any PRs submitted have a high chance of being refactored.</p>
<p>One thing that would be appreciated immediately is installing Runcible, and interacting with the API classes. Let me know how things work out, and how you think the output/public methods should be tweaked for usability and clarity.</p>
<p>I’m also more than happy to take old network equipment off of people’s hands so long as it still runs a relevant network operating system (for things that can’t be run in a VM anyways.) One of the goals that I have in regards to CI is that each build will have all of the core plugins integration tested with their respective platforms, and in some cases hardware is required.</p>