I can recommend some stuff I’ve been using myself :

• Dolibarr as an ERP + CRM : requires some work to configure initially. As most (if not all) features are disabled by default, it requires enabling them based on what you need. It also has a marketplace with a bunch of modules you can buy
• Gitea to manage codebases for customer projects. It can also do CI but I’ve not looked into it yet
• Prometheus and its ecosystem (mostly promtail and grafana) for monitoring and alerting
• docker mail server : makes it quite easy to self host a full mail server. The guides in their doc made it painless for me to configure dmarc/SPF/other stuff that make e-mail notoriously hard to host
• Cal.com as a self hostable alternative to calendly
• Authentik for single sign-on and centralized permission management
• plausible for lightweight analytics
• a mix of wireguard, iptables and nginx to basically achieve the same as cloudflare proxying and tunnels

I design, deploy and maintain such infrastructures for my own customers, so feel free to DM me with more details about your business if you need help with this

I did not read the link, but two of my biggest concerns do not appear in the summary you provided :

• the burden of hosting an ActivityPub enabled service is often duplicated for each instance instead of being split between them (for example, my Lemmy instance has a large picture folder and database because it is replicating all posts from communities I’m subscribed to)
• it’s a privacy nightmare. All instance admins now have as much spying power as the single centralized service it is replacing

(Edit: typo)

It’s not that I don’t believe you, I was genuinely interested in knowing more. I don’t understand what’s so “precious” about a random stranger’s thought on the internet if it’s not backed up with any source.

Moreover, I did try searching around for this and could not find any result that seemed to answer my question.

Can you give examples of countries where mainstream media is not owned by billionaires ?

On this day, exactly 12 years ago (9:30 EDT 1 Aug 2012), was the most expensive software bug ever, in both terms of dollars per second and total lost. The company managed to pare losses through the heroics of Goldman Sachs, and “only” lost $457 million (which led to its dissolution).

Devs were tasked with porting their HFT bot to an upcoming NYSE API service that was announced to go live less than a 33 days in the future. So they started a death march sprint of 80 hour weeks. The HFT bot was written in C++. Because they didn’t want to have to recompile once, the lead architect decided to keep the same exact class and method signature for their PowerPeg::trade() method, which was their automated testing bot that they had been using since 2003. This also meant that they did not have to update the WSDL for the clients that used the bot, either.

They ripped out the old dead code and put in the new code. Code that actually called real logic, instead of the test code, which was designed, by default, to buy the highest offer given to it.

They tested it, they wrote unit tests, everything looked good. So they decided to deploy it at 8 AM EST, 90 minutes before market open. QA testers tested it in prod, gave the all clear. Everyone was really happy. They’d done it. They’d made the tight deadline and deployed with just 90 minutes to spare…

They immediately went to a sprint standup and then sprint retro meeting. Per their office policy, they left their phones (on mute) at their desks.

During the retro, the markets opened at 9:30 EDT, and the new bot went WILD (!!) It just started buying the highest offer offered for all of the stocks in its buy list. The markets didn’t react very abnormally, becuase it just looked like they were bullish. But they were buying about $5 million shares per second… Within 2 minutes, the warning alarms were going on in their internal banking sector… a huge percentage of their $2.5 billion in operating cash was being depleted, and fast!

So many people tried to contact the devs, but they were in a remote office in Hoboken due to the high price of realestate in Manhattan. And their phones were off and no one was at their computer.

The CEO was seen getting people to run through the halls of the building, yelling, and finally the devs noticed. 11 minutes ahd gone by and the bots had bought over $3 billion of stock. The total cash reserves were depleted. The compnay was in SERIOUS trouble…

None of the devs could find the source of the bug. The CEO, desperate, asked for solutions. “KILL THE SERVERS!!” one of the devs shouted!!

They got techs @ the datacenter next to the NYSE building to find all 8 servers that ran the bots and DESTROYED them with fireaxes. Just ripping the wires out… And finally, after 37 minutes, the bots stopped trading. Total paper loss: $10.8 billion.

The SEC + NYSE refused to rewind the trades for all but 6 stocks, the on paper losses were still at $8 billion. No way they coudl pay. Goldman Sachs stepped in and offered to buy all the stocks @ a for-profit price of $457 million, which they agreed to. All in all, the company lost close to $500 million and all of its corporate clients left, and it went out of business a few weeks later.

Now what was the cause of the bug? Fat fingering human error during release.

The sysop had declined to implement CI/CD, which was still in its infancy, probably because that was his full-time job and he was making like $300,000 in 2012 dollars ($500k today). There were 8 servers that housed the bot and a few clients on the same servers.

The sysop had correctly typed out and pasted the correct rsync commands to get the new C++ binary onto the servers, except for server 5 of 8. In the 5th instance, he had an extra 5 in the server name. The rsync failed, but because he pasted all of the commands at once, he didn’t notice…

Because the code used the exact same method signature for the trade() method, server 5 was happy to buy up the most expensive offer it was given, because it was running the Sad Path test trading software. If they had changed the method signature, it wouldn’t have run and the bug wouldn’t have happened.

At 9:43 EDT, the devs decided collectively to do a “rollback” to the previous release. This was the worst possible mistake, because they added in the Power Peg dead code to the other 7 servers, causing the problems to grow exponentially. Although, it took about 3 minutes for anyone in Finance to actually inform them. At that point, more than $50 million dollars per second was being lost due to the bug.

It wasn’t until 9:58 EDT that the servers had all been destroyed that the trading stopped.

Here is a description of the aftermath:

It was not until 9:58 a.m. that Knight engineers identified the root cause and shut down SMARS on all the servers; however, the damage had been done. Knight had executed over 4 million trades in 154 stocks totaling more than 397 million shares; it assumed a net long position in 80 stocks of approximately $3.5 billion as well as a net short position in 74 stocks of approximately $3.15 billion.

28 minutes. $8.65 billion inappropriately purchased. ~1680 seconds. $5.18 million/second.

But after the rollback at 9:43, about $4.4 billion was lost. ~900 seconds. ~$49 million/second.

That was the story of how a bad software decision and fat-fingered manual production release destroyed the most profitable stock trading firm of the time, and was the most expensive software bug in human history.

never stopped POSTing, even though I configured nginx to always respond 403 to anything from them for about a year now.

Lol, there are definitely some stubborn user agents out there. I’ve been serving 418 to a bunch of SEO crawlers - with fail2ban configured to drop all packets from their IPs/CIDR ranges after some attemps - for a few months now. They keep coming at the same rate as soon as they get unbanned. I guess they keep sending requests into the void for the whole ban duration.

Using 418 for undesirable requests instead of a more common status code (such as 403) lets me easily filter these blocks in fail2ban, which can help weed out a lot of noise in server logs.

I think they do get marked as dead after the Bodis subdomain does not act as a Lemmy instance. But I was wondering if a large number of instances “waking up from the dead” and acting maliciously could cause some trouble. Or would such “undead” instances pose no more threat to the fediverse than the same number of newly created malicious instances ? I’m mainly thinking about stuff like being in a privileged position to DoS most instances at once, or impersonation of accounts that used to actually exist on these “undead” instances

It’s the clients (web/android app, probably iOS too) that are making these requests.

To the best of my knowledge, the Immich server inside the container is not making requests to the outside. It is merely sending a style.json to the client displaying a map, which then fetches tiles from the Cofractal URL inside this JSON.

Or you can quite easily configure nginx as your personal caching proxy with an arbitrarily long TTL/retention duration (you can check out my follow-up post for instructions on doing that)

I used to wonder what kind of nerd notices this kind of thing, now I’m one of them

Edit : If you want to join us :

• you can run Pi-hole which is a self-hosted DNS server that allow monitoring/blocking DNS requests from devices configured to use it. In its default configuration, it acts as a network wide ad/tracker blocker.
• On Android, you can install Rethink DNS. This will configure itself as a VPN on your device, forcing all traffic to go through it. This allows it to act as an on-device firewall that allow monitoring/blocking DNS requests and TCP/UDP connections. This is similar to the features of Pi-hole, but the fact that it’s on-device allows it to be app aware : the logs will detail which app is responsible for which connection, and the allow/block rules can be app-dependent. The app honestly goes beyond all my expectations :
  • it does a good job at being easy to use by default
  • it is very configurable which gives you a lot of control if you want/need/can handle it
  • You can configure it to route traffic (after applying firewall rules) to a Wireguard VPN or through Orbot. (Apps that act as VPNs are not compatible with each other : you can only have one active at a time)
  • You can even configure several Wireguard interfaces at the same time, and route specific apps through specific tunnels

What do you mean ? Can you give me the exact link that’s not working ?

At this point, I’ll just assume you are trolling and stop replying after this comment.

This post is trying to provide a generic solution to the fact that there are no reasonable way to get map tiles without relying on a third party provider.

I additionally included instructions on how to set it up with Immich, but I don’t see how a caching proxy in front of OSM should be part of Immich, a software focused on managing photo libraries.

How does an nginx config fit as a “diff” when the Immich repo and docker images do not include nginx (or any other reverse proxy) ?

Blocking the DNS was the first thing I did. This is intended to restore the map feature without having to trust a random company I’ve never heard of.

What do you mean by “a diff of a code fix” that would be simpler ?

You can, but you would not be able to display the map. Might as well disable the map server-wide

Not yet, but I will probably submit a PR to include this guide in the docs

I’ll try clarifying what I had in mind :

I tried running maptiler to generate tiles from OSM’s data, which required an insane amount of time and resources (not doable for most self-hosters including myself, even for a single country) to process the data and store the results. I was wondering if there would be a way to ask maptiler (or another equivalent tool) to only generate tiles that contain points from a given set (in this case, photos) and maybe the tiles adjacent to them. What about doing this for every zoom level ? This would require generating at most zoom_levels * n_photos (* 9) if we include adjacent) tiles, and a lot less for the typical person taking several photos at the same place.

Thanks for sharing your experience and for the links.

Do you think it would be doable to make/host a tileserver that only generates the first few zoom levels for the whole planet by default, and is able to generate tiles for more detailed zoom levels only for specific locations ? I’m thinking of a feature where Immich asks the tile server to generate the appropriate tiles based on the locations of photos. Since we only ever zoom on locations where photos have been taken, and we often take several photos at the same locations, could this decrease the requirements enough for self-hosting ?

No need to be rude…

Thanks for the detailed feedback. According to one Immich dev, they used to use OSM’s raster tile provider but switched away from it since they were causing too much load on OSM’s servers.

There does not seem to be any non-commercial vector-tile provider at the moment (though OSM seems to be currently working on it), and it seems really overkill to try and self-host a tile provider (at least with the default level of details). Maybe the way is to find a balanced level of details that makes it reasonable to self host