Ok, so I have heard on here that nitrogen is good for the initial growth but potassium is needed for fruit to set. My soil has a lot of chopped up plant matter in it like grass and grass roots (from the tilling)which will probably be good for the nitrogen levels. My question is how do I add potassium to the soil?

The nitrogen present in the grass etc will be used up in the decomposition process. So not really a benefit to the trees other than it is organic matter. It will actually deplete some of the existing nitrogen from the soil so you'll need to replace it.

I use for all my citrus, mango, avocado and others, a mix of 3 parts blood and bonemeal to one part sulphate of potash to 3 parts pelletised chicken manure 3 times a year around the dripline. If you haven't planted yet, you could dig the mix into the soil and let it rest for a few weeks or so.

I've found the best soil for growing fruit trees is one which has previously had a few crops of vegies thru it, as you can iron out the problems and develop the soil with the vegies and when you plant the trees it's a well balanced soil.

For example, manure a bed heavily, dig in the above mixture and grow some potatoes. Afterward, grow a crop of sweetcorn and cucurbits such as cucumber or melons. . Then, depending on the results, plant your fruit trees. The potatoes will break up the soil and you'll get a good feed. The sweetcorn will burn off all the excessive manures and give you a good indication of the quality of the soil. You can mulch around the sweetcorn heavily with compost too to increase your yield and further improve your soil.
This technique works great and though it takes a little longer, the end results outweigh the time factor. I plant my fruit trees in Autumn as where i live it's only a mild winter and they get some root growth in before winter hits.

* * * *

The secret is the soil.