That's the part that would leave me concerned.
Having my email password compromised?
That's a bit of a "meh" moment.
Suddenly discovering that one password now gave access to
potentially all my financial accounts as well?
That's a wake up in the night with cold sweats moment. :(
Just a note about security threat modeling: your email password can generally be used to reset all your other passwords, so actually having your email password compromised is one of the most terrifying situations of all. Unless, of course, you use a security key with gmail, in which case compromise of your password may not get the attacker very far. ;)
The Chrome password manager is convenient, and the sync can be incredibly handy (I can sign into stuff on different computers or even my phone without needing to copy over the passwords), but you might consider leaving your highest-value passwords out of that system, or really any system. Personally, my financial passwords are not known by Chrome, myself, or even my password manager. (Yes, you heard that right -- no single entity knows the passwords. How? By using a simple secret-splitting scheme -- I memorize part of the password, and my password manager stores the rest.)
Damian