36

Because of hardware failure we had to move some vpns from one datacenter to another.

The team of highly untrained monkeys at my hosting provider were hired to do this. First they ran backups of all the systems. Then they started the moving process. A few hours later they were done. We got an email everything was back online.

So we restarted all our processes and no data was coming in from our Raspberry's around the country. So we start a little investigation. What did these buffons do, they changed our rsa keys.

So we kindly ask them to put the old keys back so we do not have to fix 200 changed key warnings on systems that are not remotely accesible.

Apperently something that can't be done because their back up process is automated and always makes new keys.

Holy fucking fuck, whats the point in having a backup its not an exact copy. Is this fucking normal?

Now I will be spending the next few weeks literally standing in cow shit reconnecting Raspberry's.

Thanks a fucking lot. Not!

Comments
  • 2
    WTF did I just read! They don't understand what the word backup means 😓
  • 8
    @AlexDeLarge we measure power usage for a lot of dairy farmers equipment. For data collection and sending we are using Raspberry's.

    They are placed next to the equipment that needs measuring. But well the cows are there as well.

    We are using the farmers own network to save cost, but that makes it impossible to connect from the outside.

    Normaly if we do updates the Raspberry's will connect to our servers. Download the update and restarts. But well since they can't connect anymore we have to go there and switch sd cards with an update with the new keys.
  • 0
    @MisterArie what software are you using to do this?
  • 2
    @mt3o Raspberry's are running on Linux and connecting through ssh. All the data reading, converting showing are custom build.

    We are using Python, C, PHP, shell, XML, json, JavaScript.

    Data we collect mostly comes in binary or hex. Through modbus or mbus. But the amount of meters we read out is increasing so also are the formats we handle.
Add Comment