Tag: server

  • LetsEncrypt it!

    Hello, Quite a quick turnaround for blog posts and me. This one is going to be helpful for those of us that use Wildcard Certificates in our environment and found out that our SSL provider changed their policies based on industry standards, but now their certificates cost 200x more than they used to so we are moving to an opensource and free solution.

    For those of you that don’t know what the previous paragraph means, Google, and other major web site providers implemented that all communications on the Internet be secured. To do this, we use SSL or Secure Socket Layer certificates. These certificates verify and validate that the site you are on is the real one and any information that you provide on it will be encrypted and secure. SSL Certificate do this encrypting and signing to make sure everything is good. In the past, we used to have to spend $100’s, if not $1000’s (Like it did) to have this capability. LetsEncrypt came about to make this free and accessible to everyone. The downside is that the certificates are only valid for 90 days instead of a year, but you get what you pay for.

    I am moving to this model because my vendor of SSL, Digicert changed their model and now I can’t renew certificates without spending another $600 on top of the $5k I’ve already spent. So I am moving to LetsEncrypt.

    LetsEncrypt is a SSL company that uses a software package called certbot that can automatically create and install certificates that are trusted to systems.

    My DNS host provider, however, is not one of their partners. However, they do allow me to edit records on the fly, which is important since that is how LetsEncrypt verifies that you own the domain and won’t generate a certificate if you don’t. This means that I can’t automate the deployment or the generation, and I have to run the following command to update my certificates every 90 days. Some of my systems can be automated, which those, like this one that runs my web server, can. However, I do have some systems like my Virtual Center server or my Email server that use Wildcard or a single certificate to cover multiple servers. This blog will discuss how to do this, mainly so that in 90 days I can remember how to do this.

    So, on to how to do this.

    First, install certbot on a machine. Since I’m a Linux person, and I used Ubuntu, I installed this on my local machine:

    sudo apt update
    sudo apt install letsencrypt

    This installs the base LetsEncrypt software with no plugins. Since my DNS provider does not have a plugin, I have to do this manually.

    I also had to add the wildcard, or “*” to my domain to prove I owned the domain, so I logged in to my DNS provider, and created an “A” record that pointed to my webserver with the *.lucaswilliams.net name. This will allow me to use this certificate on any of my server inside my lucaswilliams.net domain. Very useful for virtual server for VMware, email, and other servers that need HTTPS and SSL Certificates.

    Once I created the wildcard domain entry in my DNS record. I then went to the terminal on my Linux machine and typed the following:

    sudo certbot certonly --manual \
    --preferred-challenges=dns \
    --email user@domain.com \
    --server https://acme-v02.api.letsencrypt.org/directory \
    --agree-tos \
    -d *.domain.com

    for the --server https://acme-v02.api.letsencrypt.org/directory line, you have to use this server to create the certificate as this is the only one that LetsEncrypt uses to for this requirement.

    After hitting enter to start the process, I was presented a prompt asking if I wanted to share my information and details about the certificate, which I replied “N” but if you want you can.

    The next prompt is the important one. It looks like the following:

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Please deploy a DNS TXT record under the name:
    
    _acme-challenge.domain.com.
    
    with the following value:
    
    q12yr1dyFyrh143HHRTe42HH_hf#1d7&ewftgs8H
    
    Before continuing, verify the TXT record has been deployed. Depending on the DNS
    provider, this may take some time, from a few seconds to multiple minutes. You can
    check if it has finished deploying with aid of online tools, such as the Google
    Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.domain.com.
    Look for one or more bolded line(s) below the line ';ANSWER'. It should show the
    value(s) you've just added.
    
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Press Enter to Continue

    I then logged into my DNS provider and created this TXT record and then hit enter in the terminal to complete the key generation. LetsEncrypt verified the record and created my certificates in /etc/letsencrypt/live/domain.com. You will need ‘root’ access for the live directory, so I ran sudo -i to change to root user and access the certificates.

    I then copied the privekey.pem and fullchain.pem files to my servers and renamed them to what the system understands as the private key file and the certificate file.

    The biggest takeaway from this is that LetsEncrypt will let us create certificates for our systems for free, however, they only are valid for 90 days which means that we have to do this every 3 months. Some systems can be fully automated if their names don’t change or the certificate is used only for validation and verification of the site and name, but more complex certificates, like email verification and signature signing, this is the way to go for that.

    If any of you know of a better way of doing this, please let me know and I’ll share it and give you credit for the improvment!

  • Replacing failed disks in a MD RAID in Ubuntu Server

    Hello everyone! Been a moment since my last blog update. This is a special one that I have been wanting to write, but wanted to wait until I actually had to do it so I can show real world examples, and boy, is this one for the record books.

    So, my secondary KVM server has a 5 disk hot swappable chassis that I bought on NewEgg about 7 years ago that allows you to install 5 SATA disks and these disks are connected to the mother board from the chassis into the 5 SATA ports. This allows me to hot swap the hard drives if they ever fail, and well, two of them did about a month ago. The system is setup as a RAID-5. So all of the disks are members of the RAID and then the 5th disk is a Hot Spare. Well, Disk 4 and 5 failed together. Basically, disk 4 failed, and while 5 was becoming the 4th disk, it failed. Luckily the Array was still good, but now I need to replace the failed disks.

    I bought 2 new 2TB disks from NewEgg and installed them in the array. Unfortunately, the system does not automatically detect new drives installed or removed, so I had to run the following commands to get the disks recognized by the system.

    sudo -i
    echo "0 0 0" >/sys/class/scsi_host/host0/scan
    echo "0 0 0" >/sys/class/scsi_host/host1/scan
    echo "0 0 0" >/sys/class/scsi_host/host2/scan
    echo "0 0 0" >/sys/class/scsi_host/host3/scan

    I then listed the /dev/ directory to make sure that /dev/sdd and /dev/sde were no longer being seen as they have been removed. I also checked the raid configuration to make sure that they were not listed any longer:

    mdadm -D /dev/md0
    mdadm -D /dev/md1

    Both arrays no longer listed the failed disks, so I’m ready to physically add the new disks.

    I installed the new disks. Now I need to re-scan the bus for Linux to see the disks:

    echo "0 0 0" >/sys/class/scsi_host/host0/scan
    echo "0 0 0" >/sys/class/scsi_host/host1/scan
    echo "0 0 0" >/sys/class/scsi_host/host2/scan
    echo "0 0 0" >/sys/class/scsi_host/host3/scan

    I then listed the /dev directory and I can now see the new disks, sdd and sde.

    I then need to make sure that they have the correct format and partition layout to work with my existing array. For this I used the sfdisk command to copy a partition layout and then apply it to the new disks:

    sfdisk -d /dev/sda > partitions.txt
    sfdisk /dev/sdd < partitions.txt
    sfdisk /dev/sde < partitions.txt

    If I do another listing of the /dev directory I can see the new drives have the partitions. I’m now ready to add the disks back to the array:

    mdadm --add /dev/md0 /dev/sdd2
    mdadm --add /dev/md1 /dev/sdd3
    mdadm --add-spare /dev/md0 /dev/sde2
    mdadm --add-spare /dev/md1 /dev/sde3

    I then check the status of the array to make sure it is rebuilding:

    mdadm -D /dev/md0
    mdadm -D /dev/md1

    The system shown it was rebuilding the arrays and at the current rate it was going to take about a day.

    The next day I go and check the status, and low and behold I found out that disk 5 (sde) had failed and was no longer reporting in. I got a bad disk shipped to me. So I contacted NewEgg and they sent me out a replacement as soon as I sent them the failed disk. Luckily it was the hot spare so it didn’t have any impact on the system removing it or adding it back, but I did run the following command to remove the spare from the array and then re-scanned the bus so that the disk was fully removed from the server:

    sudo mdadm --remove /dev/md0 /dev/sde2
    sudo mdadm --remove /dev/md1 /dev/sde3
    sudo echo "0 0 0" >/sys/class/scsi_host/host0/scan
    sudo echo "0 0 0" >/sys/class/scsi_host/host1/scan
    sudo echo "0 0 0" >/sys/class/scsi_host/host2/scan
    sudo echo "0 0 0" >/sys/class/scsi_host/host3/scan
    sudo mdadm -D /dev/md0
    sudo mdadm -D /dev/md1

    The MDADM reported that there was no longer a spare available and the listing of the /dev directory no longer shown /dev/sde. A week later, I got my new spare from NewEgg and installed it and ran the following:

    sudo -i
    echo "0 0 0" >/sys/class/scsi_host/host0/scan
    echo "0 0 0" >/sys/class/scsi_host/host1/scan
    echo "0 0 0" >/sys/class/scsi_host/host2/scan
    echo "0 0 0" >/sys/class/scsi_host/host3/scan
    ls /dev
    sfdisk /dev/sde < partitions.txt
    ls /dev
    mdadm --add-spare /dev/md0 /dev/sde2
    mdadm --add-spare /dev/md1 /dev/sde3
    mdadm -D /dev/md0
    mdadm -D /dev/md1

    This added the disk and then added it as a hot spare for the arrays. Since it’s a hot spare, it does not need to resync.

    And there you have it, how to replace the disks in a MD RAID on Ubuntu.

  • Minecraft Server for Ubuntu 20.04.2

    Hello everyone. I hope you are all doing well. I am writing this blog entry because I created a Minecraft server for my kids some time ago, but I had a hardware failure in the system and never replaced it. At the time, it was no big deal since the boys decided that they were done with Minecraft. But lately, with this new version of Minecraft, they have gotten back into it, and they wanted to have a shared sandbox that they can play with their friends on.

    So, I rebuilt their Minecraft server, but this time, I did it from 16.04 to 20.04. It was pretty straight forward and not much has changed in the way of doing this, but this is here for those of you that want to deploy your own Minecraft server.

    NOTE: This will only work for the Java version of Minecraft. If you are using the Windows 10 version or the one on Xbox or Switch, you will not be able to connect to this server.

    So, the first thing you need is a clean installation of Ubuntu 20.04.2 Server. The system specs should be at least 4GB of RAM and 2 CPU Cores and 80GB of Storage. After you install Ubuntu, do the normal first boot practices, update, upgrade, reboot if required, etc.

    sudo apt update && sudo apt upgrade -y

    Once that is completed, you need to install a couple things on top.

    One thing I like is the MCRcon tool from Tiiffi. I use this to do backups and statistics of my server, and it is really easy to use, and it’s super small. So I install the Build-Essential package as well as git. Minecraft also leverages Java, so I install the Open Java Development Kit packages with headless mode:

    sudo apt install git build-essential openjdk-11-jre-headless

    Once that is completed, I then create a minecraft user so that when I run this as a service, it is a lot more secure, and I have a location where to keep all the dedicated Minecraft files.

    sudo useradd -m -r -U -d /opt/minecraft -s /bin/bash minecraft

    This creates the Minecraft user with the home directory in /opt/minecraft. This also doesn’t create a password for this account so we don’t have to worry about someone gaining access to our system with this account. You can only access this account via sudo su - minecraft with your local admin account.

    Now, we need to switch to the minecraft user and run the following:

    sudo su - minecraft
    mkdir -p {server,tools,backups}
    git clone https://github.com/Tiiffi/mcrcon.git ~/tools/mcrcon
    cd ~/tools/mcrcon
    make
    

    This will create the required directories for Minecraft, and download and build the MCRcon tool. You can verify that the MCRcon tools built successfully by running the command:

    ~/tools/mcrcon/mcrcon -v

    You will get the following output:

    mcrcon 0.7.1 (built: Mar 26 2021 22:34:02) - https://github.com/Tiiffi/mcrcon
     Bug reports:
         tiiffi+mcrcon at gmail
         https://github.com/Tiiffi/mcrcon/issues/

    Now, we get to installing the Minecraft Server Java file.

    First, we need to download the server.jar file from Minecraft. You can go here to download the file, or what I did, is I go to the link, and from there, I right click the link and select ‘Copy Link Address’ so I can paste it into my terminal on the server and use wget to install it.

    wget https://launcher.mojang.com/v1/objects/1b557e7b033b583cd9f66746b7a9ab1ec1673ced/server.jar -P ~/server 

    Now, we need to run the Minecraft server. It will fail on the first run because we need to accept the EULA. We also need to modify the server.properties file since the first run creates these files:

    cd ~/server
    java -Xmx1024M -Xms1024M -jar server.jar nogui

    After the program fails to start, we need to modify the eula.txt file and change the eula=false at the end of the file to eula=true. Save this file and exit.
    Next, we need to enable RCON in Minecraft. Open the server.properties file and search for the following variables, and change them accordingly:

    rcon.port=25575
    rcon.password=PassW0rd
    enable-rcon=true

    Also, while you are in this file, you can make any other changes that you want to the server, such as the server name, listening port for the server, the MOTD, etc. Also, choose a complex password so that not just anyone can remote control your server.

    Now, I like to have this run as a service using SystemD. To do this, create a service script. First you have to exit as the Minecraft user by typing exit and getting back to your local admin shell. Then run the following:

    sudo vi /etc/systemd/system/minecraft.service

    Paste the following in the file:

    [Unit]
    Description=Minecraft Server
    After=network.target
    
    [Service]
    User=minecraft
    Nice=1
    KillMode=none
    SuccessExitStatus=0 1
    ProtectHome=true
    ProtectSystem=full
    PrivateDevices=true
    NoNewPrivileges=true
    WorkingDirectory=/opt/minecraft/server
    ExecStart=/usr/bin/java -Xmx2G -Xms2G -jar server.jar nogui
    ExecStop=/opt/minecraft/tools/mcrcon/mcrcon -H 127.0.0.1 -P 25575 -p PassW0rd stop
    
    [Install]
    WantedBy=multi-user.target

    Save the document. Next, run

    sudo systemctl daemon-reload

    This will refresh systemd with the new minecraft.service.

    Now, you can start the minecraft service:

    sudo systemctl start minecraft

    To get it to start on reboots, execute the following:

    sudo sytemctl enable minecraft

    The last thing we have to do is create the backup job for your server. This uses the MCRcon tool and crontab to clean up the server as well.

    Switch back to the Minecraft user and perform the following:

    sudo su - minecraft
    vi ~/tools/backup.sh

    Paste the following script into the new file you are creating:

    !/bin/bash
     function rcon {
       /opt/minecraft/tools/mcrcon/mcrcon -H 127.0.0.1 -P 25575 -p PassW0rd "$1"
     }
     rcon "save-off"
     rcon "save-all"
     tar -cvpzf /opt/minecraft/backups/server-$(date +%F-%H-%M).tar.gz /opt/minecraft/server
     rcon "save-on"
     # Delete older backups
     find /opt/minecraft/backups/ -type f -mtime +7 -name '*.gz' -delete

    Now, create a crontab to run the backup:

    crontab -e
    0 0 * * * /opt/minecraft/tools/backup.sh

    Now exit as the Minecraft user and return as the local admin. Lastly, because I leverage UFW for my firewall, I need to open the port to the world so that people can connect to it. I do that with the following commands:

    sudo ufw allow from 10.1.10.0/24 to any 25575
    sudo ufw allow 25565/tcp
    

    This allows the Remote console to be accessed only by my internal network, and allows the Minecraft game to be accessed by the outside world.

    Now, you are ready to connect your Minecraft clients to your server and have some fun!

    Let me know if this guide worked for you or if you have any questions or comments, please leave them below.

  • Setting up Unreal Tournament 2004 Game server on Ubuntu 16.04

    Hey everybody.

    Been a while since I wrote here. Figured I would write up a howto to setting up a Unreal Tournament 2004 server. I really love this game. It brings back tons of memories, playing this when I was in the Navy with my friends on the sub.

    My boys have some break time off from school, and they played a little bit back in the day, so I decided to spin up a server so that we could play. I looked for a way to do this online, and couldn’t find anything so I figured I would write something up, so here you go.

    So, the good thing is that because the game is pretty old now, over 13 years old now, it doesn’t really require a lot of CPU or memory or storage. I deployed a KVM with 2 cores and 4GB of RAM and 20GB storage server running Ubuntu 16.04.3, and got all the updates installed. I then spent the next few hours searching for the ut2004 dedicated server package. Never could find it. Luckily, I had a backup copy, which I have uploaded to this server so you can download it here. You’ll also need the patch, which you can download here.

    I created a directory for the game in /usr/local/games/UT2004 and extracted the .zip here:

    sudo unzip -d /usr/local/games/UT2004 dedicatedserver3339-bonuspack.zip

    Once that was complete, I then untarred the patch and had to manually install it, since it creates a directory called UT2004-Patch so I had to actually go into each directory and move the files into their respective directories in the UT2004 directory. Once that was complete, you now have a system capable of running Unreal Tournament 2004 server. However, I needed to do a couple more things.

    Next, you need to install libstdc++5 package. This is required so that Unreal can run. Run the following command to install libstdc++5:

    sudo apt install libstdc++5

    One, I decided to start the web admin. In the /usr/local/games/UT2004/System/UT2004.ini. Find the UWeb.Webserver section and modify it:

    [UWeb.WebServer]
    Applications[0]=xWebAdmin.UTServerAdmin
    ApplicationPaths[0]=/ServerAdmin
    Applications[1]=xWebAdmin.UTImageServer
    ApplicationPaths[1]=/images
    bEnabled=True
    ListenPort=80

    You can change the ListenPort to what ever you want, you just need to change bEnabled=False to True to enable it.

    Next, I decided that I wanted this to run as a service using SystemD instead of just running in the background with me logged in to the server. Below is my UT2004-Server.service file:

    [Unit]
    Description=Unreal 2004 Dedicated Server
    After=network.target
    
    [Service]
    Type=simple
    User=ut2004
    WorkingDirectory=/usr/local/games/UT2004/System
    ExecStart=/usr/local/games/UT2004/System/ucc-bin-linux-amd64 server CTF-BridgeOfFate?game=XGame.xCTFGame?AdminName=admin?AdminPassword=XXXXXXXX ini=UT2004.ini log=server.log -nohomedir
    Restart=on-abort

    Just change the ?AdminPassword= to what you want I then copied the file into /lib/systemd/system and chmod 644 and chown root:root the ut2004-server.service file and now I can control the service with systemctl:

    systemctl start ut2004-server.service and I can get status with systemctl status ut2004-server.service

    One last thing I did as well is I included my cdkey from my game since I was getting errors about a missing cdkey, however, I have tested it, and it is not required. The game will still run, you just can’t advertise your server on the Internet and host Internet games without it, which means your stats also won’t work. You used to be able to download a CD-Key from Epic, but that service is no longer working. I emailed them about this on December 2, 2017 with no reply as to date.

    Happy gaming!