Firefox 130 is bringing a game-changing feature: automatic alt-text generation for images using a fully private on-device AI model! 🙌🏾
Initially available in the built-in PDF editor, our aim is to extend this to general browsing for screen reader users. hacks.mozilla.org/2024/05/expe…
Experimenting with local alt text generation in Firefox Nightly - Mozilla Hacks - the Web developer blog
Firefox 130 will feature an on-device AI model that automatically generates alt-text for images, integrated into its built-in PDF editor.Tarek Ziadé (Mozilla Hacks - the Web developer blog)
Julia :v_bi: likes this.
reshared this
AlexTECPlayz
in reply to Mozilla • • •This is how it's done! Private, open-source AI models running locally.
Q: How much storage do the models take? (EDIT: 200MB according to the post - yeah, in this case, this better be a downloadable 'module' instead of being built-in) Could you make this feature optional, which would require the user to opt-in and download or delete the model(s) themselves? I don't want Firefox to go the Microsoft Edge route, where they shovel every feature under the sun, the user has no choice, and there is no way to reduce the storage occupied by the browser.
__ol
in reply to AlexTECPlayz • • •@alextecplayz Just wait until you hear about Mozilla's brand new shopping toolbar. They bought a company that used to dabble in NFTs before switching to claiming to have AI.
And just for fun, this new Mozilla subsidiary will sell browsing history and location data to advertisers... as laid out here.
fakespot.com/privacy-policy
(Ctrl+F for "Personal Information is Sold")
Fakespot - Love Everything You Buy
www.fakespot.comKrista, Darth Moose Shark reshared this.
Matthias
in reply to Mozilla • • •Looks interesting at first glance.
Thanks for being this open and transparent about the process, used model etc.
d@nny mc²
in reply to Mozilla • • •s92
in reply to d@nny mc² • • •@hipsterelectron @hannah I like how you're yelling at one of the few organizations that go to great lengths to build a completely independent browser that they "don't give a shit" about the free web.
If they don't give a shit then literally nobody on earth does.
d@nny mc²
in reply to s92 • • •d@nny mc²
in reply to d@nny mc² • • •Emi
in reply to d@nny mc² • • •That project has a different goal than Mozilla's alt-text AI and I am sure you can use both - human descriptions with that project for the few images that will have it and Mozilla's AI for the rest.
d@nny mc²
in reply to Emi • • •d@nny mc²
in reply to d@nny mc² • • •Emi
in reply to d@nny mc² • • •@hipsterelectron @sasha92 There are 3.2B images uploaded in a day (1.2T /year), many of them are repeating, google has 130B indexed. You can't describe all of that. Sure, human description will probably be better in many cases, but AI descriptions are still very useful.
Also, I doubt that project will get as many people editing it as Wikipedia has, so it can be great for a few popular images, memes, etc. but it can never cover random images on social media and websites without alt text.
d@nny mc²
in reply to Emi • • •Tom Ritchford
in reply to Mozilla • • •Nice work!!!
This would be particularly useful for postings to Mastodon, where alt-text is much socially desirable.
peacememories
in reply to Mozilla • • •Akseli :quake_verified: :kde:
in reply to Mozilla • • •hm, i think this can be useful, however the problem is when people will never look at the output and just accept it at face value.
Basically I hope you will add a warning box that says "Do note that the text generation is not perfect and you should make sure the text clearly fits the image" or something along those lines. Also when it generates the text, it should always add "This alt text was generated by Firefox language model." as the first sentence, so people who rely on alt text features will know that this may be inaccurate.
Rainer Zufall
in reply to Mozilla • • •morgan
in reply to Mozilla • • •Fahri Reza 🍉
in reply to Mozilla • • •Danil
in reply to Mozilla • • •Is it time to invent webbrowser that actually dont spy on you?
maybeanerd
in reply to Mozilla • • •would be amazing if this offered an API for webapps to use. E.g. mastodons alt text field could detect it has that feature available and provide a suggested alt text to users.
On the other hand this might encourage lower quality alt texts, as that will always be quicker to do than writing down your own alt text.
Maybe keeping it "fallback" only for consumers of content that is missing alt text is best.
Michal Bryxí 🌱
in reply to Mozilla • • •Afferand
in reply to Mozilla • • •Justin
in reply to Mozilla • • •Emma
in reply to Mozilla • • •joene 🏴🍉
in reply to Mozilla • • •rachael laura arbor :tranarchy_punk_nonbinary:
in reply to Mozilla • • •no. get this garbage out of here
Kaito
in reply to Mozilla • • •Condalmo.
in reply to Mozilla • • •Kai und der Andere
in reply to Mozilla • • •Tushar Chauhan
in reply to Mozilla • • •excited for the mastodon rise
in reply to Mozilla • • •go follow @azutsune@wetdry.world
in reply to Mozilla • • •T.J. Crowder
in reply to Mozilla • • •GrayGooGlitch :v_lesbian:
in reply to Mozilla • • •Danielle Pond
in reply to Mozilla • • •hapax
in reply to Mozilla • • •Ænðr E. Feldstraw
in reply to Mozilla • • •skua
in reply to Mozilla • • •What a shitty idea.
Wish I was surprised that Mozilla has jumped on the AI Highway to Hell.
But their priorities have not been user focused for years IMO.
youtube.com/watch?v=4hhlQU0zDp…
AC/DC - Highway to Hell (Live - from Countdown, 1979)
YouTubeScott D. Strader 😐
in reply to Mozilla • • •Methuselah
in reply to Mozilla • • •Dream Hollow
in reply to Mozilla • • •Automated alt-text isn't too bad. This is a fair use of AI that doesn't really step on any toes.
I could foresee this causing problems if the alt-text is very wrong, though.
ikt 🇺🇦
in reply to Mozilla • • •filobus
in reply to Mozilla • • •"The first time the user adds an image, they’ll have to wait a bit for downloading the model (which can take up to a few minutes depending on your connection) but the subsequent uses will be much faster"
Hope you can disable this feature and 200 mb download at all
I understand it can be very useful for some users, but for me not at all
If I find porno images I don't understand I think I can skip them 🤪
ikt 🇺🇦
in reply to Mozilla • • •ITT all 7 firefox users upset that AI might be useful for some things
Not pictured: 2 billion people using Chrome
Florian
in reply to Mozilla • • •sll - un kien avec un capiau
in reply to Mozilla • • •Smart move, Mozilla.
Dźwiedziu
in reply to Mozilla • • •I'm still waiting for the feature that obligatory speaks back the comment to the user before posting.
(see xkcd.com/481/)
Listen to Yourself
xkcdHarrison Totty
in reply to Mozilla • • •Copernicron
in reply to Mozilla • • •Martin Rocket
in reply to Mozilla • • •“The image shows a birthday cake with lit candles in the foreground and a smiling woman in the background, likely in a room with several people.”
That's indeed longer than the Firefox text but not absurdly lengthy and detailed.
However, I'm impressed that Firefox does that locally.
Pier-Luc Brault
in reply to Mozilla • • •Marcus Adams
in reply to Mozilla • • •Simon Lucy
in reply to Mozilla • • •cameronbosch :endeavourOS:
in reply to Mozilla • • •F4GRX Sébastien
in reply to Mozilla • • •Mmmm
in reply to Mozilla • • •Petesmom
in reply to Mozilla • • •hacknorris
in reply to Mozilla • • •Steven Goetz 🇨🇦
in reply to Mozilla • • •srg
in reply to Mozilla • • •`Da Elf
in reply to Mozilla • • •I don't actually Want that in my browser.
If I did, I suppose I'd run, I don't know, Fucking Windows?
Confrontation Jacen
in reply to Mozilla • • •Hunterrules
in reply to Mozilla • • •Hannah Kolbeck 🏳️⚧️
in reply to Mozilla • • •- Founder, Alt-Text.org
A feral Cass (they/them)
in reply to Hannah Kolbeck 🏳️⚧️ • • •0x5DA
in reply to Hannah Kolbeck 🏳️⚧️ • • •i'm not mozilla, but i am interested to know more. because you worry AI alt-text would be too low quality, and be detrimental?
Hannah Kolbeck 🏳️⚧️
in reply to 0x5DA • • •@0x5DA I don't have capacity in this moment for the full depth, but it's a bit more complex. Writing alt text that's actually equalizing of access, especially on social media, requires knowledge of multiple layers of context in which an image appears. Similar to other AI types, AI description works impressively *sometimes* but falls down hard on many types of image commonly appearing on SM, often in ways not obviously bad to those writing alt text.
An example: bsky.app/profile/hannah.the-vo…
Hannah is probably online 🟢 (@hannah.the-void.social)
Bluesky Social0x5DA
in reply to Hannah Kolbeck 🏳️⚧️ • • •@hannah
hm, i see. and i guesss you don't think is a training thing..
is it worse than _no_ alt text?
a human will have a superior understanding of context, and should be strongly preferred -but many people don't all the same, and i'm doubtful that will change. this seems (from an outside perspective) to be a reasonable, if unsatisfying, solution.
Hannah Kolbeck 🏳️⚧️
in reply to 0x5DA • • •@0x5DA It can be useful for a person using a screen reader to have access to an AI description, but crucial there is that said user needs to know that that's the source of said description. There are repeated patterns of those who feel pressured to include descriptions but don't actually care about accessibility doing the absolute minimum, manifesting here as using the direct AI output without examination or editing.
So yes, a lack of inline alt text is better than AI gen inline.
dilbert 1
in reply to Hannah Kolbeck 🏳️⚧️ • • •> said user needs to know that that's the source of said description.
this is a browser feature that the end user turns on or off themselves so yes they do know that. it's not being done by the publisher.
Hannah Kolbeck 🏳️⚧️
in reply to dilbert 1 • • •dilbert 1
in reply to Hannah Kolbeck 🏳️⚧️ • • •Hannah Kolbeck 🏳️⚧️
in reply to dilbert 1 • • •Jamie is a friendly nut
in reply to Mozilla • • •Javielico 🌖
in reply to Mozilla • • •Florian
Unknown parent • • •Chrome and Edge have had this feature for a while now, sans LLM, and really the only time that is useful is when there's text in an image, which gets OCR'ed ...relatively ... well. So in that sense I can see it; PDFs often are pictures of text and this might bridge that divide. For practically any other purpose though ...no, probably not :)
Luna Lactea
in reply to Mozilla • • •Chris
in reply to Mozilla • • •Carlos Francisco 🦣
in reply to Mozilla • • •afrangry
in reply to Mozilla • • •Tofu Musubi
in reply to Mozilla • • •Jcrabapple
in reply to Tofu Musubi • • •simon.old
Unknown parent • • •Going by my own experiences using LLMs from OpenAI, Anthropic, and Google to describe images, there is never a scenario where I would rather have no description instead of one generated with AI, and I expect things to get better from here. Maybe a few people will be less inclined to describe their images if a browser can do it for them, but not everyone uses that browser, and I would guess most sighted people who are aware of alt text won't really know or care about the specifics of one browser's implementation of image descriptions. People either do or do not post alt text. If anything, maybe this announcement will make *more* people post alt text. Here on Mastodon, most people know what it is already, but I bet people are reading this post and thinking "Oh, I should probably describe my stuff so the AI doesn't do it worse."
Janie Karma S 🏳️🌈🖖🏿
in reply to Mozilla • • •Sensitive content
Krishna Draws✏️
in reply to Mozilla • • •Ben Zanin
in reply to Mozilla • • •Kent Ahrens
in reply to Mozilla • • •Ferrous
in reply to Mozilla • • •Bifurkatus 🎭
in reply to Mozilla • • •My photo collection search could use such a thing.
#digikam
Joe Cooper 🇺🇦
in reply to Mozilla • • •Thomas Dorr
in reply to Mozilla • • •Frank Heijkamp
in reply to Mozilla • • •Frank Heijkamp
Unknown parent • • •Hannah Kolbeck 🏳️⚧️
Unknown parent • • •@paper @sasha92 I want to say explicitly that I don't fully agree w/ @hipsterelectron here, but I think that your dismissal of their concerns also substantially misses the mark
I've also talked at length about fine details that assist the Blind folks @weirdwriter talks about in using AI above while not incurring discussed harm. Making it easy for a screen reader user to knowingly get an AI description is relatively simple and worthwhile, but giving that tool to writers needs great care
100rabh™
in reply to Mozilla • • •me_valentijn
in reply to Mozilla • • •These modules need to be opt-in from the beginning ... not mandatory with a "maybe we'll let you get rid of them some day, teehee" that seems to be your current plan.
I REALLY don't want that functionality, and especially not the bloat or other resource use. Guess it's time to start looking for a new browser.
Lambda :neofox_flag_nb:
in reply to Mozilla • • •kira5w
in reply to Mozilla • • •Dsens
in reply to Mozilla • • •Muelsyse
in reply to Mozilla • • •Dysphoric Enbyboy
in reply to Mozilla • • •rubyda
in reply to Mozilla • • •Robert A.Mason
in reply to Mozilla • • •Mwa 🇸🇾🇵🇸
in reply to Mozilla • • •