And I think it's far worse when you do it early in the loan period.
In 2009 I took on a 30 year mortgage, at 4.75% interest. I could not afford the larger payment for a shorter mortgage, and so was fine with what I had. Several years later, as interest rates dropped, I did some casual inquiry into refinancing. Every query resulted in costs that made refinancing make no sense.
Recently I did refinance, and I'm almost wondering if I didn't outsmart myself. There were no up front costs to me, I got a 3.75% interest rate, but because the refinance was to roll the cost of my solar panels (yes! I went solar!) into the mortgage, I owe more than I did before. Which I suppose makes sense. And the refinance was for 20 years, the same as the remaining term on the original mortgage. Right now I am able to put $300 additional each month into the mortgage payment, which will take nearly ten years off the term of the loan and save me a boatload of interest.
Perhaps keeping your current mortgage and making extra payments would work for you. Here's a link to a calculator to help figure that out: https://www.calcxml.com/calculators/extra-payment-calculator
If for some reason that doesn't seem to work for you, you can google extra mortgage payments and find other calculators out there.