I have not seen exactly this in Mathematica but I think being able to solve an equation set for only one unknown may be supported.
The direct way would be of course to implement the inverse matrix algorithms and have them stop when the particular element is determined. It would probably be interesting to see what optimizations you can have for each algorithm if the goal is to get one element only.
But according to the first paragraph, one idea could be to express your matrix inverse as unknowns in equation system and then try solving only for the unknown you need (since solving equation set for a single unknown should be supported). If you get better memory and CPU performance then it may be safe to say that Mathematica did not actually solve the whole equation but just did its optimizations to determine only your unknown. You may get worse performance of course since you are not using inverse algorithms directly. If you decide to try let us know if it worked!