The notification is influenced by the Travel Time in macOS Sonoma. The easiest way is to completely remove the Travel Time and the notification will always show up 10 min before, at 6:20PM.
You can find more details about the Travel Time behaviour in the Apple Support docs.
The time to leave is calculated based on the preference you choose in
the Maps app in the View > Directions menu.Calendar finds your starting location by using the location in any
events that are up to 3 hours before this event. If Calendar doesn’t
find a location, it uses your work address or home address (from your
card in the Contacts app), depending on the time of day. (Set your
work hours in Calendar General settings using the “Day starts at” and
“Day ends at” pop-up menus.) If your card in Contacts doesn’t have
your addresses, Calendar uses your computer’s current location.
Determining your computer’s location requires that Location Services
be turned on (see Change Privacy settings).You receive a notification based on your likely location before the
event starts, the event’s location, and current traffic or transit
conditions before you need to leave, when you need to leave, and if
you’re running late.