Duplicati - Amazon Cloud Drive - Cert Errors

Creating this thread off the details of THIS thread. Topic wasn't accurate, wanted to bring more attention to a thread outlined with the actual issue in Subject line.

Problem is related to Duplicati and it's underlying Mono install being able to properly import the right root certificates to trust for SSL communications. The problem, that I've encountered, is continually having my backups error out or fail to start, due to a certificate mismatch on the server side. Knowing Amazon likely changes front end servers for CloudDrive, the cert hashes keep changing. The main problem being, that the top level root isn't trusted, so it's looking for some sort of validation that it's ok to trust these lower sub certs due to the chain not having a trusted root. (I Believe that's the issue at least)

@ryecoaaron has been nice enough to help troubleshoot by making adjustments to the plugin to test out various workaround or resolutions. The first was adding in the mozroots solution as outlined by many to solve this in a normal Debian based installation. That didn't seem to do the trick. After this, he altered to include the other suggested workaround of adding an Environment Variable to specify MONO_TLS_PROVIDER=legacy. At first, I thought this seemed to work, but no luck after running for awhile, I returned to the same error again.

Any ideas folks have? Has anyone been able to successfully get Duplicati running on other systems with the same issue? What was done to fix/work around this issue? Any other alternatives to what can be done to the OMV plugin to adjust/account for scenarios outside of the Debian standard install?

@ryecoaaron - honestly I'm surprised nobody else is having this issue. Or perhaps folks got fed up like I almost did and just turned on the ACCEPT ALL SSL option. Anyway, I wanted to dig into this for the good of others and hopefully share the details around what's going wrong for me and what I think may be the solution as I know you don't use the plugin.

The error being returned is:
Failed to connect: The server certificate had the error RemoteCertificateChainErrors and the hash A8A56350F7BEEFDE0F6074159E7EB0DA745577BE
BTW - the main issue that keeps happening, is that once you accept this hash once, it works for awhile. Then the servers rotate on some regular basis, and the next connect fails due to a changed hash from a different front-end certificate with the same root that isn't trusted.

The full error output is:
Failed to connect: The server certificate had the error RemoteCertificateChainErrors and the hash A8A56350F7BEEFDE0F6074159E7EB0DA745577BE If you trust this certificate, use the commandline option --accept-specified-ssl-hash=A8A56350F7BEEFDE0F6074159E7EB0DA745577BE to accept the server certificate anyway. You can also attempt to import the server certificate into your operating systems trust pool. You may want to import a set of trusted certificates into the Mono certificate store. Use the command: cert-sync /etc/ssl/certs/ca-certificates.crt #for Debian based systems cert-sync /etc/pki/tls/certs/ca-bundle.crt #for RedHat derivatives Read more: http://www.mono-project.com/docs/about-mono/releases/3.12.0/#cert-sync

So with some basic reserch, I come to understand that mono doesn't by default come with any trusted roots. It relies on the user to initiate or choose the trusted roots you'd like to use. The commonplace way to go is the mozroots command. It appears this pulls trusted roots from Mozilla. Without digging too much, I'm not sure what realistically is pulled from this (though I'd imagine similar to the browser). For some reason though, Mono still has an error appear with a cert in the chain being presented by Amazon in the Duplicati app. Thus the error.

Based on a quick read of the documentation following the link supplied, mozroots is now a legacy way to go and not necessarily a forward looking supported option. BUT, they do offer up the cert-sync tool as a new option that is used to pull the local trusted roots from the host. This detail can be read at the link in the error message: HERE. It may require certain packages being installed/available inside the Debian install before functioning properly if not installed from a "mono-project.com package". I didn't follow your code enough to identify if the package is pulled and built manually, pulled from a specific source, or other.

The previous suggestion of the Environment Variable, seems to actually point to a way to avoid using the NEWER TLS modules. Which could in fact cause further issue in the future as I'm sure soon enough AMZ will stop allowing older TLS models. So is it possible to make some changes to the plugin to a test repo or test version I can manually load to test out? Rather than having the plugin continually updated without knowing if it works, I'd be happy to help work and test if it's functioning properly.

I'm also looking into testing making a basic connection from the console to begin with to see if I can get connected to the Amazon Cloud Drive servers from the server. This will at least validate/verify if it is in fact Mono having the issue, or if something else somewhere could somehow be amiss.

Zitat von ryecoaaron

It needs the mono-project repos for a new enough version of the package.

My assumption (read, my foolish understanding ) is that this requires more of a mess than just being able to support the built in repos that are being used within Debian/OMV already? Adding another repo I would assume just adds more complication with dependencies, potential to screw up other packages, and the like?

Are you familiar with Mono at all? I ask as I'm completely dumb to it and never dealt with it. To be honest, I'm in the weeds and dangerous enough to find myself lost in linux systems mucking with configs and doing some basic understanding to slightly reverse engineer things, but I'm in no way a coder or guru with all the different moving parts of apps.

Would there be a way that I can validate using mono commands what roots exist? What roots have been pulled in using mozroots perhaps? Or even a way to try and manually run mozroots just to validate it is working properly?

Ok, so did some digging and came up with a few pieces of information, and validation/questions. It does seem at a high level this isn't just affecting OMV users, it's affecting a wide breadth of folks, but they're likely using the recommended solutions, which don't seem to be working here. Or they're using the accept-any-ssl option, which just isn't secure at all.

Duplicati Wiki - SSL/TLS support - this page outlines the details around the many issues folks are seeing. Using their TlsTest.exe tool, I can validate/verify that I'm seeing the CERT_E_CHAINING errors they outline at the top of the page. Reading down there are some solutions for various different platforms. My understanding, OMV 3.0 which I'm running, is based on Debian 8 Jesse. The instructions here say what I indicated earlier, to use the newer packages. I know you said that's not the best solution, so I'd like to work through alternatives before giving in that I can't get it to work as is.

So I continued down seeing the option for the cert-sync line, which is only included in the newer releases of Mono. It seems that using this older version of Mono for Duplicati, is the core problem, as the SSL/TLS support was not as strong compared to current versions. The line about attempting to use the MONO_TLS_PROVIDER=legacy was in there, which obviously didn't do the trick either. I validated with DPKG that the mono packages specified below it are installed, they are. I also attempted using the idea in the Ubuntu 14.04 section with mozroots, outlining to wget the list manually in case there was an actual issue grabbing via SSL since SSL is the problem. No luck.

Validation - I believe you had said in the other thread that Duplicati does run as Root correct? I just want to validate that we don't have an issue that it's running as a user that isn't importing the certs with mozroots. I also tried using the --machine option for mozroots to install into the machine trust store found in /usr/share/.mono/certs/Trust - no luck here either. Now I could be grabbing at straws here, but one odd thing I did see, is when I run mozroots manually from the CLI, I get an output of:

Importing certificates into user store...
5 new root certificates were added to your trust store.
Import process completed.

If I run the mozroots command again, it outputs the same thing saying 5 new roots were added. Which makes me wonder if there is some sort of issue with these 5 roots. Though looking at them I doubt they're the actual issue (UserTrust Network, DigiNotar, Secure Digital). So what I'm starting to think, is if there is a way to run a different (newer) version of Mono in OMV? I know I had seen the addons with different repos for Mono, or is that what you were trying to get away from completely? It seems to me if we were to run this under newer version of Mono, the whole issue should go away as it would have cert-sync packaged and/or come with the ca-certificates-mono package.

Got it. Ok, I have a large backup processing again, so I'm going to wait until it's finished to test this out so I don't screw anything up. Hopefully it'll be done before the week is out. So it may be a few days until I have feedback.

Totally understand the desire to get away from extra repos. I wish I knew mono and apps a bit better to be able to try and extract the cert-sync tool for porting over and using so it could hopefully just remedy the situation. That sounds like it should work, but I'm not aware of the dependencies and such. Or I wonder if it's possible I could just use the project repo to get the cert-sync tool and run it, then revert back to base repo version? I'd imagine the certs are stored in the same location, irrespective of version? Maybe I can push the idea on the Duplicati app side to pull the cert-sync tool and dependencies in just like they did with the mozroots and a few others in the utility folder.

Zitat von ryecoaaron

That sounds like a bad idea.

I like your optimism! ... but I do agree.

Ya I figured it would be long winded and chaotic, but for testing it may be a path to identify what's missing. I think being able to find a way to mimic the cert-sync function is the best answer. Based on my basic understanding, it seems like mozroots and cert-sync are just producing a list of the certs in a specific format that mono is looking for. My guess is cert-sync is literally just processing all the certs in the host machine SSL Trust directory, and doing a large batch job of converting the certs to the proper format for Mono to read/understand. Just by looking at the roots on the base OMV install, I can see there are MANY more certs than the directory from mozroots. However, I also surmise that the ca-cartificates file in the machine level SSL Trust directory contains most of what is in the mozroots import, with perhaps a few more.

It's likely I'm going to be looking at what the formats are, and getting a copy of the transposed certs to identify what is missing from mozroots that we need. If I can isolate those, then it could be a matter of creating a basic script to do SSL conversion of the host level certs. This is what I'm imagining cert-sync is doing at the end of the day. Hopefully we can just target a few specifics though for this so it's speedy and can just target the specifics on the host.

So finally got around to testing this out. Added the Mono repo. Installed the ca-certificates-mono package. Looks like it worked, but now when I try to run cert-sync I get an error:

Mono Certificate Store Sync - version 5.0.0.0
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.


Error: System.TypeLoadException: Could not resolve type with token 0100001f (from typeref, class/assembly Mono.Security.Interface.MonoTlsProviderFactory, Mono.Security, Version=4.0.0.0, Culture=neutral, PublicKeyToken=0738eb9f132ed756)
  at Mono.Tools.CertSync.Main (System.String[] args) [0x00022] in <7c7506fc8d25432eb32c48fd3f510701>:0

Google wasn't friendly in returning anything that provided any value. Looks like maybe some type of mismatch on the versions of Mono.Security? Could this simply be due to the version of mono in the debian repos vs mono-project repos? Any easy way to resolve without having to F up my mono instal?

I stopped trying with it as I temporarily got around it as indicated (not so secure manner). The larger issue was around mono libraries and the plugin not using newer libraries I believe. For me, I ended up totally dumping it anyhow since Amazon no longer allows use of Cloud Drive for unlimited storage at a fixed cost. So I'm still on the hunt for a new alternative where I can make the most economical decision - thus I'm not using Duplicati at this point.