Simrad AP 26 autopilot interfaced with new Garmin GPS

Take the boat out and put the autopilot in setup mode, swing/calibrate the compass on the autopilot, then do the seatrial for the autopilot. See if that helps.

Where are the compasses installed. Anything new around the AP26 compass?

Swing the compass as Skippy J above recommends after you make sure nothing (lost screw driver, other tool, new Garmin compass, new wires) is around the AP26 compass.

Just remembered something, Did you by chance ad a N2K wind device?

In auto mode, the AP26 uses the Simrad autopilot compass for heading and that is it. Should not be getting any heading information from the Garmin for steering the boat.

The AP26 RobNet uses the Simrad RC36 rate compass.
The AP16, 25, 26 & 27 were the first to offer SimNet and N2K I/O while still using the RobNet base systems. Sadly, the N2K group prioritys were not set up very well at the beginning of N2K and even worse for the SimNet/crossover. The AC10,20&40 box may see the N2K compass signal as the preferred signal but not at the proper comms rate.
If the newer Garmin compass or weather vane compass is tied to the N2K back bone, the AC box will drive itself crazy figuring out what compass to talk to.
Not a problem on the Ray E120 sense the SeaTalk 2, N2k and old SimNet were of the old stuff and not using the higher anarchy of the latest N2k.

When in doubt, after swinging the compass and problems persist, disconnect the AP from the N2K back bone and try the pilot in auto mode.

Here is where old preparatory N2K and later standards clash (CRASH) together. The selling shop should of known this of the N2K old/new buss. You probably will respond that you have a Garmin compass and maybe a weather head. The Garmin compass is needed to overlap radar on the charts. Sometimes the weather head compass will work here also.
The fast fix, Go back to NMEA 183 from the Garmin to the AP box both ways and remove the AP from the SimNet/N2k back bone. Or turn off your new money compass options from your wind vane or Garmin compass.

Or, The new Garmin smart pilot is pretty cool. If you enjoy throwing money at somebody, this would be the ultimate. Touch the steering wheel and the pilot turns it self off and your driving in a panic scenario..

Check out the prices of used AP25-26 equipment on e-bay may help.

Once you fix your problem(s), kick the ass hole who talked you into the Garmin equipment.

What your simple scenario does not register, near three different generations of serial network coms. The Garmin with the new and most complete and open N2k talking to a very old and incomplete SimNet just barely N2K comparable AND RobNet, Simrads own option network. Group priorities, option IDs and standardized option timing was not complete in the old days (10 to 15 years ago). The AP25 family of products is 15+ years old.
A modern Garmin option is a compass that allows radar overlay over charts that can not be completed with GPD direction. Also, Airmar offers a N2K weather head that is very common with new equipment with a mini compass to detect wind direction. Some sounders need a compass for recording Bathy data.

So, Until we really get a complete inventory list from the O P and after he completes some compass calibrations, Were just guessing.

I have been here and just went thru it again, My old AP25 (same as the O P) just was not talking well with all my other stuff.
It took a while to get my New Simrad AP, Simrad AIS 400, Simrad RS35 radio, Simrad IS20 displays, Simrad WR-WB20, old Raymarine E120 plotters with Ray DSM, RNS & Ray compass talking to my large screen PC. Interconnects using SeaTalk, SeaTalk2 (old N2K) Ray H-S(eithernet), USB, N2K, SimNet and N183/RS232.

Been here, done this, consumed mass quantities..

That is what I have been trying to say. 15+ years age between the new Garmin and the old Simrad AP. May not be possible to do. So then, you need another compass.
I had the same AP. I needed a Ray Compass for my MFD to get my charts & radar to overlay.
The Simrad RC36 compass is on RobNet and does not do well on SimNet nor with its 15 year old technology talk well with todays N2k.

There are so many possibilitys of errors when putting old and new stuff together. I've just explained what troubles I have had and observed on other installations.

The O P stated he has a new plotter and radar mfd.
If he has spent these big bux, I assume he would want radar over laid on his charts.
Here you usually need a compass for the mfd.
The O P has yet to confirm his whole package.

Further, There is technical issues with the older RobNet AP compass and the new N2K I/O between the AP and new Garmin equipment that may not allow the AP compass to be used in the Garmin.

The OP stated he had an E120 MFD. I had the same package on my boat (AP25 & E120) and I needed then a second compass for my E120s radar / chart overlay. I am guessing he had and kept this second compass or installed a new Garmin compass for his new mfd.

Usually, when people spend the bux on equipment like this, the little extras are just pennies extra like an Airmar weather station, that includes it's own compass and I have witnessed these compasses indues errors on mfds.
Again, without further notes from the OP, I'm just expanding on what I'm guessing at.

Swinging all of his equipment needs to be done first we all agree. Disconnect or unplug the AP from all other equipment would be next in line for trouble shooting.

The LightHouse stuff is of a newer generation of equipment. N2K grouping has come a long way in 15 years.
More flexible and smarter set up options also.