Smart. Wild Wasn
Steinberger doubles down on a pet theme of his: inference quality through intermediary providers is a lottery, and anything that bakes consistency into the default is overdue. He has a point — the same model behind different proxies can behave noticeably differently, since quantization, routing and context handling all leak into outputs.… more
Aggregators sell convenience and fallbacks, but for those of you shipping production apps the price has long been unpredictability, which is exactly why he advised going straight to the source. If this becomes the industry norm, middlemen will need a much better justification for their cut than 'we route stuff'.
This is smart. Wild that this wasn’t the default. Unpredictable inference quality was usually why I recommended skipping the middleman.
Source:
X — Peter Steinberger
Discussion (0)