I run dnstools.ws which lets you perform DNS lookups, pings, traceroutes, etc. from 25 locations around the world. Each location is powered by a VPS running Debian, running a C# service that’s compiled to native code ahead-of-time using Native AOT. It uses ~60MB RAM.

Six of the the locations are powered by tiny “NAT VPSes” (native IPv6 with shared NAT IPv4) that only cost a few dollars a year, sponsored by various server providers. These usually have 256 MB RAM and 4-5 GB disk space.

This is great with OpenVZ and LXC. Since they’re containers that share the kernel with the host, kernel memory doesn’t count towards the container’s memory limit. I’m using ~75 MB RAM on those systems: ~60MB for the DNSTools worker and ~15MB for everything else (sshd, systemd, cron, rsyslogd, and unattended-upgrades). Plenty of room left.

I also have a few KVM systems with 256 MB RAM. These are what I’m struggling with.

Debian 13 (Trixie) increased the minimum hardware requirements from 256 MB to 512 MB RAM. It seems like this is a hard requirement - When running on a system using 256 MB RAM, the installer complains about having too little RAM, and OOMs during the installation. Even with a successful installation (e.g. upgrading from bookworm to trixie), it kernel panics on boot: “System is deadlocked on memory”.

I could try debootstrap to bootstrap a basic system, or Clonezilla to clone a working disk image over the network, but I think I’d hit the memory deadlock too.

Does Debian have smaller kernel images for VM environments, that use less RAM? Or should I just give up on Debian for this use case?

Does anyone have a recommendation for another distro I should use? I’ve been considering trying Alpine. C# does support compiling to use musl instead of glibc, so that’s not an issue. I’m also not tightly-coupled to systemd and can get rid of it.

I can mount a custom ISO on the systems, so booting from an ISO isn’t an issue.

Thanks!

Edit: Alpine looks very promising - no issue installing it and running my app on a 256MB VM. This is probably what I’ll end up using.

  • utopiah@lemmy.ml
    9·
    3 hours ago

    Alpine in container is typically considered the smallest one can conveniently use (not going through LinuxFromScratch or writing your own OS). I did some tinkering a while back ending up with 14.29MiB memory footprint.

    • dan@upvote.auOP
      3·
      2 hours ago

      In my case it needs to be a VM rather than a container (because that’s what the hosting company offers), but Alpine is looking promising so far. No issues with booting from the ISO and installing it on a system with 256MB.

      I got my app running on Alpine too. Now I just need to update my Ansible playbook to handle Alpine, and do more thorough testing. Will look into it later in the week.

  • adarza@lemmy.caEnglish
    10·
    4 hours ago

    tui installer on netinst iso (13.4) says 320mb is required:

    This system does not have enough memory to be supported by this program. At least 320 megabytes of memory are required. If you continue, unpredictable and broken behavior will likely result. You should reboot now and add memory or find some other program to use.

    it does allow you to continue anyway, which switches to ‘low memory’ expert mode install. 320mb also gives this error. 384mb does not, but still enters the special install mode, where i was able to load enough installer components to set up networking to start an install. it did lose part of the display font partway through but it did finish and reboot ok (with ~104mb used). dropped the ram in virt-man to 256mb and it still booted.

    edit: i just installed dietpi on this, which basically shreds debian down to the kernel and not much else, and rebuilds it as dietpi. went on without any issues, not even a complaint with 256mb. htop says 65mb used (this, before adding any server ‘apps’, mind you)… but i might just be able to upgrade a couple dietpi boxes i have that do only have 256mb ram.

    • dan@upvote.auOP
      1·
      3 hours ago

      Interesting! I wonder why I was hitting the memory deadlock on boot, but yours booted OK.

      I might run a local VM, use dietpi, then clone the resulting VM to the hosting provider using Clonezilla.

      • adarza@lemmy.caEnglish
        2·
        3 hours ago

        why I was hitting the memory deadlock on boot, but yours booted OK

        i dunno. was your debian-installer running in lowmem mode? did it add extra stuff because of the detected vm environment (i did notice at least one extra thing here for qemu plus the network drivers needed).

        this trixie-based dietpi here boots at 192mb allocated to the vm, but panics at 160mb.

        the dietpi x64 ‘installer’ iso don’t use the debian-installer. they are clonezilla-based, preconfigured to use an included dietpi image, which does a first-run setup after it’s ‘restored’.

        • dan@upvote.auOP
          1·
          3 hours ago

          I was using debian-installer in lowmem mode, but it OOMd during the install. I haven’t tried adding swap yet though.

          The Trixie system that I did manage to setup was an upgrade from Bookworm. That’s the one that’s hitting the memory deadlock on boot.

          • adarza@lemmy.caEnglish
            1·
            3 hours ago

            maybe see if the providers can bump the ram up a bit to get through the trixie installer? 384mb was enough here (i didn’t try to install with less), perhaps 512mb in your hosts’ environments will be enough.

  • non_burglar@lemmy.world
    6·
    4 hours ago

    You can get creative with Linux.

    1. Install on 512mb, remove and trim system services, and lower the memory. Running KVM is by itself likely not going to let you do that, I suspect.
    2. Use Alpine as you say.I run many alpine containers on between 25MB and 60MB, I give them 256MB, but it’s way overkill.

    Regarding alpine, be prepared to find differences from glibc and systemd distros in places you don’t expect. PHP (god forbid you should need it) is a right mess on alpine. Mongodb will not work on alpine. Stuff like that.

    I may take some challenge on this, but the tcpip stack seems faster on alpine than in debian, at least in my use cases.

  • mlfh@lm.mlfh.orgEnglish
    10·
    5 hours ago

    The grub command line options at the beginning of this article might help get your system booted without the memory deadlock, and then you can make further adjustments as needed: https://tierhive.com/blog/tierhive-howto/debian-13-minimal-guide-reduce-ram-to-38mb-and-disk-to-275mb

    Alpine is great for exactly this kind of thing, though, and I use it often in embedded environments where resources are at a premium. Just do some good reading up on it beforehand, since can be very different if you’re used to debian and systemd.

    • dan@upvote.auOP
      5·
      4 hours ago

      Thanks for the link!

      I’m trying Alpine locally in a VM with 256MB RAM, and so far so good. I got my app successfully cross-compiled using musl-gcc, rsync’d it over, and it starts on the VM with no issues. Now I just need to figure out all the stuff around it (like certbot) and do some more thorough testing. I use an Ansible playbook to deploy to the Debian servers, so I’ve got to update it to handle Alpine too.

  • QueenMidna@lemmy.caEnglish
    1·
    3 hours ago

    I wonder how much smaller you could get it by just cross-compiling to WASM and using some of the edge hosts out there

    • dan@upvote.auOP
      1·
      2 hours ago

      That’s an interesting idea that I didn’t consider. .NET does seem to have some support for WebAssembly.

      Many of the current systems were provided by various hosts for free though, which is how I expanded to so many locations. The 256MB RAM systems are only a few dollars per year, so those hosts were happy to provide a few for free.

      • QueenMidna@lemmy.caEnglish
        1·
        43 minutes ago

        Yeah I run an application on fly.io and they don’t even bother charging me anything because my WASM app takes so few resources

  • agentTeiko@piefed.socialEnglish
    6·
    5 hours ago

    I mean you could just compile your own kernel from the Debian source build and use the build menu to pull out everything you don’t need to get the memory requirements down. Then bootstrap with your kernel.

    • mlfh@lm.mlfh.orgEnglish
      10·
      5 hours ago

      The ongoing maintenance for this would be a bit of a pain, since you’d need to recompile every update on a separate machine with enough memory to do so, package it up into a .deb, and distribute and install it everywhere.

      I do this on a little raspberry pi cluster and it works, but it’s work.

    • dan@upvote.auOP
      4·
      5 hours ago

      I’ve considered this, it just might be a pain to keep up to date with kernel updates. I guess I could create an appropriate config and then automate the builds.

      I also forgot that Debian has a cloud-specific kernel (linux-image-cloud-amd64) which excludes a bunch of drivers. I’ll try that out too.

      The other thing on my list to try is mmdebstrap with a basic Debian install, using runit or openrc instead of systemd.

    • dan@upvote.auOP
      3·
      5 hours ago

      Even if I avoid the installer (e.g. by cloning an existing system, or by installing an older version of Debian then upgrading it), and have swap space, the kernel hits a memory deadlock on boot. I suspect it’s deadlocking before swap is initialized.

      • Ralph Rönnquist@mastodon.au
        1·
        4 hours ago

        @dan @RIotingPacifist A “normal” debian starts by unpacking the initrd, which is into ram, so you’d need enough to cover that, plus initial runtime. Most likely you will need to tailor your initrd; i.e. go non-standard.

        • dan@upvote.auOP
          1·
          3 hours ago

          That’s what I was thinking. I might try the cloud kernel (linux-image-cloud-amd64). It only has drivers required for VM platforms, so maybe the initramfs might be smaller? Otherwise I could build a custom one with just the things I need (only ext4 and swap, only drivers for KVM, etc).

          I’m trying Alpine as well, which looks promising.

  • Munkisquisher@lemmy.nz
    3·
    5 hours ago

    Tinycore is a 23mb Linux desktop that runs in 46mb of ram. Microcore is the headless version that runs in 28mb of ram.

      • zener_diode@feddit.org
        5·
        4 hours ago

        Yes, TinyCore-Core comes without a GUI (like the commenter above said, headless). When they say 28 MB RAM, that might even be overestimating it a bit. All of TinyCore-Core fits into just 17 MB. Of course that goes up if you need to install things, and I don’t know if it comes with an ssh-Server out-of-the-box. So for a proper server setup, 28 MB sounds pretty reasonable.

        http://tinycorelinux.net/

        It blows my mind, that there is an entire Linux Distro, that fits into a modern processor’s cache. Who needs RAM anyways? (unfortunately, because of how cache works, we can’t actually use it without RAM)

  • rbn@sopuli.xyz
    1·
    4 hours ago

    Another thought: is it required for your service to physically run in 25 locations? Couldn’t you have one (more powerful) main instance which accesses the other servers via SSH to execute whatever command is required?

    • dan@upvote.auOP
      3·
      3 hours ago

      The 25 locations are a bare minimum Debian installation with just my app, SSH (for management), Certbot (for TLS certs), and a few other utilities (ping, traceroute, mtr) added. The app uses ~60MB RAM so it’s not an issue.

      The site accesses each location using gRPC over a persistent connection.

      The code is open-source if you’re curious: https://github.com/Daniel15/dnstools

    • dan@upvote.auOP
      1·
      5 hours ago

      I’ve installed Debian Buster, Bullseye and Bookworm on these systems from the netinst ISO, with no tweaking needed. Trixie is the first release where I’ve been unable to do this, and even upgrading the system doesn’t work (hits a memory deadlock on boot).