The idea is having tensor acceleration built into SoCs for portable devices so they can run models locally on laptops, tablets and phones.
Because, you know, server-side ML model calculations are expensive, so offloading compute to the client makes them cheaper.
But this gen can’t really run anything useful locally so far, as far as I can tell. Most of the demos during the ramp-up to these were thoroughly underwhelming and nowhere near what you get from server-side services.
Of course they could have just called the “NPU” a new GPU feature and make it work closer to how this is run on dedicated GPUs, but I suppose somebody thought that branding this as a separate device was more marketable.
EU should introduce regulation that prohibits client-side AI/ML processing for applications that require internet access. Show the cost upfront. Let’s see how many people pay for that.
It’s definitely weird that everyone is panicking about data center processing costs but not about the exact same hardware powering high end gaming devices that have skyrocketed from 100W to 450W in a few years, but ultimately if you want to run a model locally you can run a model locally. I’m not sure how you’d regulate that, it’s just software.
Hell, I don’t even think distributing the load is a terrible idea, it’s just that the models you can run locally in 40 TOPS kinda suck compared to the order of magnitude more processing you get on modern GPUs.
I’m not talking about stable diffusion or anything like that.
I meant whatever Twitter, or any similar chatbots, or AI assistant features of apps should be run on server-side, not put a load on customers’ devices.
Yeah, no, I get the spirit of the thing. I’m just saying that… well, for one that it wouldn’t be a bad idea if it worked, it just doesn’t at the moment. But more importantly that regulations don’t work like that. You can’t just make rules that go “hey you guys specifically have to run this software on a server specifically”. You can already run assistants locally using a whole bunch of downloadable models, it’d be a huge overreach to tell people and companies that they CAN make the software and run it, but only remotely. That’s just… not how rules and regulations are put together.
The idea is having tensor acceleration built into SoCs for portable devices so they can run models locally on laptops, tablets and phones.
Because, you know, server-side ML model calculations are expensive, so offloading compute to the client makes them cheaper.
But this gen can’t really run anything useful locally so far, as far as I can tell. Most of the demos during the ramp-up to these were thoroughly underwhelming and nowhere near what you get from server-side services.
Of course they could have just called the “NPU” a new GPU feature and make it work closer to how this is run on dedicated GPUs, but I suppose somebody thought that branding this as a separate device was more marketable.
EU should introduce regulation that prohibits client-side AI/ML processing for applications that require internet access. Show the cost upfront. Let’s see how many people pay for that.
That is a weird proposal.
It’s definitely weird that everyone is panicking about data center processing costs but not about the exact same hardware powering high end gaming devices that have skyrocketed from 100W to 450W in a few years, but ultimately if you want to run a model locally you can run a model locally. I’m not sure how you’d regulate that, it’s just software.
Hell, I don’t even think distributing the load is a terrible idea, it’s just that the models you can run locally in 40 TOPS kinda suck compared to the order of magnitude more processing you get on modern GPUs.
I’m not talking about stable diffusion or anything like that.
I meant whatever Twitter, or any similar chatbots, or AI assistant features of apps should be run on server-side, not put a load on customers’ devices.
Yeah, no, I get the spirit of the thing. I’m just saying that… well, for one that it wouldn’t be a bad idea if it worked, it just doesn’t at the moment. But more importantly that regulations don’t work like that. You can’t just make rules that go “hey you guys specifically have to run this software on a server specifically”. You can already run assistants locally using a whole bunch of downloadable models, it’d be a huge overreach to tell people and companies that they CAN make the software and run it, but only remotely. That’s just… not how rules and regulations are put together.