Firefox 130 is bringing a game-changing feature: automatic alt-text generation for images using a fully private on-device AI model! 🙌🏾
Initially available in the built-in PDF editor, our aim is to extend this to general browsing for screen reader users. https://hacks.mozilla.org/2024/05/experimenting-with-local-alt-text-generation-in-firefox-nightly/
Experimenting with local alt text generation in Firefox Nightly - Mozilla Hacks - the Web developer blog
Firefox 130 will feature an on-device AI model that automatically generates alt-text for images, integrated into its built-in PDF editor.Tarek Ziadé (Mozilla Hacks - the Web developer blog)
like this
reshared this
AlexTECPlayz
in reply to Mozilla • • •This is how it's done! Private, open-source AI models running locally.
Q: How much storage do the models take? (EDIT: 200MB according to the post - yeah, in this case, this better be a downloadable 'module' instead of being built-in) Could you make this feature optional, which would require the user to opt-in and download or delete the model(s) themselves? I don't want Firefox to go the Microsoft Edge route, where they shovel every feature under the sun, the user has no choice, and there is no way to reduce the storage occupied by the browser.
__ol
in reply to AlexTECPlayz • • •@alextecplayz Just wait until you hear about Mozilla's brand new shopping toolbar. They bought a company that used to dabble in NFTs before switching to claiming to have AI.
And just for fun, this new Mozilla subsidiary will sell browsing history and location data to advertisers... as laid out here.
https://www.fakespot.com/privacy-policy
(Ctrl+F for "Personal Information is Sold")
Fakespot - Love Everything You Buy
www.fakespot.comGrrrr, Darth Moose Shark reshared this.
Matthias
in reply to Mozilla • • •Looks interesting at first glance.
Thanks for being this open and transparent about the process, used model etc.
d@nny "disc@" mcClanahan
in reply to Mozilla • • •alt text cannot be automatically generated without human input because the function of alt text is highly contextual. if you actually gave a shit about a freer more independent web you'd support projects like @hannah's distributed alt text database which is currently supported via browser extension. it looks like you're too late now to take it over but if you have any funding left for non-AI bullshit i'm sure the 501(c)(3) would absolutely love your support as well as built-in browser integration https://social.alt-text.org/@hannah/112522294652904233
Hannah Kolbeck 🏳️⚧️
2024-05-29 03:39:18
sasha
in reply to d@nny "disc@" mcClanahan • • •@hipsterelectron @hannah I like how you're yelling at one of the few organizations that go to great lengths to build a completely independent browser that they "don't give a shit" about the free web.
If they don't give a shit then literally nobody on earth does.
d@nny "disc@" mcClanahan
in reply to sasha • • •d@nny "disc@" mcClanahan
in reply to d@nny "disc@" mcClanahan • • •Emi
in reply to d@nny "disc@" mcClanahan • • •That project has a different goal than Mozilla's alt-text AI and I am sure you can use both - human descriptions with that project for the few images that will have it and Mozilla's AI for the rest.
d@nny "disc@" mcClanahan
in reply to Emi • • •d@nny "disc@" mcClanahan
in reply to d@nny "disc@" mcClanahan • • •Emi
in reply to d@nny "disc@" mcClanahan • • •@hipsterelectron @sasha92 There are 3.2B images uploaded in a day (1.2T /year), many of them are repeating, google has 130B indexed. You can't describe all of that. Sure, human description will probably be better in many cases, but AI descriptions are still very useful.
Also, I doubt that project will get as many people editing it as Wikipedia has, so it can be great for a few popular images, memes, etc. but it can never cover random images on social media and websites without alt text.
d@nny "disc@" mcClanahan
in reply to Emi • • •Mémoriser : Gauche Droite
in reply to Mozilla • • •Tom Ritchford
in reply to Mozilla • • •Nice work!!!
This would be particularly useful for postings to Mastodon, where alt-text is much socially desirable.
peacememories
in reply to Mozilla • • •Aks :quake_verified: :kde:
in reply to Mozilla • • •hm, i think this can be useful, however the problem is when people will never look at the output and just accept it at face value.
Basically I hope you will add a warning box that says "Do note that the text generation is not perfect and you should make sure the text clearly fits the image" or something along those lines. Also when it generates the text, it should always add "This alt text was generated by Firefox language model." as the first sentence, so people who rely on alt text features will know that this may be inaccurate.
Rainer Zufall
in reply to Mozilla • • •morgan
in reply to Mozilla • • •Fahri Reza
in reply to Mozilla • • •Danil
in reply to Mozilla • • •Is it time to invent webbrowser that actually dont spy on you?
maybeanerd
in reply to Mozilla • • •would be amazing if this offered an API for webapps to use. E.g. mastodons alt text field could detect it has that feature available and provide a suggested alt text to users.
On the other hand this might encourage lower quality alt texts, as that will always be quicker to do than writing down your own alt text.
Maybe keeping it "fallback" only for consumers of content that is missing alt text is best.
Michal Bryxí 🌱
in reply to Mozilla • • •Afferand
in reply to Mozilla • • •Justin
in reply to Mozilla • • •Emma
in reply to Mozilla • • •joene 🍉 :ecoan: :ancom: :bij1_flag: :antifa:
in reply to Mozilla • • •tree-shaped rachael
in reply to Mozilla • • •no. get this garbage out of here
Kaito
in reply to Mozilla • • •Lunatic at Large 🇨🇦️
in reply to Mozilla • • •Condalmo.
in reply to Mozilla • • •Kai und der Andere
in reply to Mozilla • • •Tushar Chauhan
in reply to Mozilla • • •excited for the mastodon rise
in reply to Mozilla • • •surchaw
in reply to Mozilla • • •Zead (also known as forsaken)
in reply to Mozilla • • •T.J. Crowder
in reply to Mozilla • • •GrayGooGirl :v_lesbian: :1up:
in reply to Mozilla • • •Danielle Pond
in reply to Mozilla • • •hapax
in reply to Mozilla • • •Ænðr E. Feldstraw
in reply to Mozilla • • •skua
in reply to Mozilla • • •What a shitty idea.
Wish I was surprised that Mozilla has jumped on the AI Highway to Hell.
But their priorities have not been user focused for years IMO.
https://www.youtube.com/watch?v=4hhlQU0zDpA
AC/DC - Highway to Hell (Live - from Countdown, 1979)
YouTubeSchneckbert 🐌
in reply to Mozilla • • •Scott D. Strader
in reply to Mozilla • • •Methuselah
in reply to Mozilla • • •Dream Hollow
in reply to Mozilla • • •Automated alt-text isn't too bad. This is a fair use of AI that doesn't really step on any toes.
I could foresee this causing problems if the alt-text is very wrong, though.
ikt 🇺🇦
in reply to Mozilla • • •filobus
in reply to Mozilla • • •"The first time the user adds an image, they’ll have to wait a bit for downloading the model (which can take up to a few minutes depending on your connection) but the subsequent uses will be much faster"
Hope you can disable this feature and 200 mb download at all
I understand it can be very useful for some users, but for me not at all
If I find porno images I don't understand I think I can skip them 🤪
ikt 🇺🇦
in reply to Mozilla • • •ITT all 7 firefox users upset that AI might be useful for some things
Not pictured: 2 billion people using Chrome
Florian
in reply to Mozilla • • •james
in reply to Florian • • •@zersiax but do we really want to give some who can’t be bothered, a check box that generates confusing, shallow and often innacurate alt text that would be more aggravating than not having any alt text at all?
This is not an “ai sucks” comment with not foundation. Artists have been using AI alt text on Instagram for a while now and it is truly awful.
Florian
in reply to james • • •Chrome and Edge have had this feature for a while now, sans LLM, and really the only time that is useful is when there's text in an image, which gets OCR'ed ...relatively ... well. So in that sense I can see it; PDFs often are pictures of text and this might bridge that divide. For practically any other purpose though ...no, probably not :)
james
in reply to Florian • • •@zersiax mastodon web and some Fediverse clients like Ivory already have that OCR, yet I still see tonnes of image posts of text that do not bother to use it.
Which is why I’m like “please don’t just launch this and expect people to check what comes out, otherwise you’ve just made experience a special sort of crap”
😬
sll against genocide
in reply to Mozilla • • •Smart move, Mozilla.
Dźwiedziu
in reply to Mozilla • • •I'm still waiting for the feature that obligatory speaks back the comment to the user before posting.
(see https://xkcd.com/481/)
Listen to Yourself
xkcdHarrison Totty
in reply to Mozilla • • •Copernicron
in reply to Mozilla • • •Martin Rocket
in reply to Mozilla • • •“The image shows a birthday cake with lit candles in the foreground and a smiling woman in the background, likely in a room with several people.”
That's indeed longer than the Firefox text but not absurdly lengthy and detailed.
However, I'm impressed that Firefox does that locally.
Pier-Luc Brault
in reply to Mozilla • • •Nobody باچیز नास्ति (he/him)
in reply to Mozilla • • •OddOpinions5
in reply to Mozilla • • •People
I have a really radical idea
Please don't flame me until you read it !!
Lets see how this works for a week or so before we decide if it is good or bad
Could be that 50% of the time this feature works, which would save everyone a lot of time
Matheus Marinho
in reply to Mozilla • • •The Pale Moon Project homepage
www.palemoon.orgMarcus
in reply to Mozilla • • •Osma A
in reply to Mozilla • • •Simon Lucy
in reply to Mozilla • • •cameronbosch :endeavourOS:
in reply to Mozilla • • •F4GRX Sébastien
in reply to Mozilla • • •Mmmm
in reply to Mozilla • • •Petesmom
in reply to Mozilla • • •hacknorris
in reply to Mozilla • • •Steven Goetz 🇨🇦
in reply to Mozilla • • •srg
in reply to Mozilla • • •`Da Elf
in reply to Mozilla • • •I don't actually Want that in my browser.
If I did, I suppose I'd run, I don't know, Fucking Windows?
thejacenallen
in reply to Mozilla • • •Hunterrules
in reply to Mozilla • • •Hannah Kolbeck 🏳️⚧️
in reply to Mozilla • • •- Founder, Alt-Text.org
Cass (they/them)
in reply to Hannah Kolbeck 🏳️⚧️ • • •0x5DA
in reply to Hannah Kolbeck 🏳️⚧️ • • •i'm not mozilla, but i am interested to know more. because you worry AI alt-text would be too low quality, and be detrimental?
Hannah Kolbeck 🏳️⚧️
in reply to 0x5DA • • •@0x5DA I don't have capacity in this moment for the full depth, but it's a bit more complex. Writing alt text that's actually equalizing of access, especially on social media, requires knowledge of multiple layers of context in which an image appears. Similar to other AI types, AI description works impressively *sometimes* but falls down hard on many types of image commonly appearing on SM, often in ways not obviously bad to those writing alt text.
An example: https://bsky.app/profile/hannah.the-void.social/post/3ktzlwoqdkc2h
Hannah is probably online 🟢 (@hannah.the-void.social)
Bluesky Social0x5DA
in reply to Hannah Kolbeck 🏳️⚧️ • • •@hannah
hm, i see. and i guesss you don't think is a training thing..
is it worse than _no_ alt text?
a human will have a superior understanding of context, and should be strongly preferred -but many people don't all the same, and i'm doubtful that will change. this seems (from an outside perspective) to be a reasonable, if unsatisfying, solution.
Hannah Kolbeck 🏳️⚧️
in reply to 0x5DA • • •@0x5DA It can be useful for a person using a screen reader to have access to an AI description, but crucial there is that said user needs to know that that's the source of said description. There are repeated patterns of those who feel pressured to include descriptions but don't actually care about accessibility doing the absolute minimum, manifesting here as using the direct AI output without examination or editing.
So yes, a lack of inline alt text is better than AI gen inline.
Angry Sun
in reply to Hannah Kolbeck 🏳️⚧️ • • •> said user needs to know that that's the source of said description.
this is a browser feature that the end user turns on or off themselves so yes they do know that. it's not being done by the publisher.
Hannah Kolbeck 🏳️⚧️
in reply to Angry Sun • • •Angry Sun
in reply to Hannah Kolbeck 🏳️⚧️ • • •Hannah Kolbeck 🏳️⚧️
in reply to Angry Sun • • •@sun @0x5DA I misunderstood their plans, and have written a partial retraction: https://social.alt-text.org/@hannah/112570794161740101
Hannah Kolbeck 🏳️⚧️
2024-06-06 17:13:22
Jamie is a friendly nut
in reply to Mozilla • • •Javielico 🌖
in reply to Mozilla • • •james
in reply to Mozilla • • •whilst I admire the desire to make accessibility easier and more present, instagram and similar have auto alt text and it is absolutely rubbish. The most famous users of it do not check it at all. My favourite artists will allow “possible three panel illustrated graphic comic of a green snake or reptile having a coversation” alongside their image
We must fund alt text education and have suitable controls and alerts as to the usefulness of the generated content, alongside bringing these “features” into the world.
james
in reply to james • • •Simon Jaeger
in reply to james • • •Going by my own experiences using LLMs from OpenAI, Anthropic, and Google to describe images, there is never a scenario where I would rather have no description instead of one generated with AI, and I expect things to get better from here. Maybe a few people will be less inclined to describe their images if a browser can do it for them, but not everyone uses that browser, and I would guess most sighted people who are aware of alt text won't really know or care about the specifics of one browser's implementation of image descriptions. People either do or do not post alt text. If anything, maybe this announcement will make *more* people post alt text. Here on Mastodon, most people know what it is already, but I bet people are reading this post and thinking "Oh, I should probably describe my stuff so the AI doesn't do it worse."
Frank Heijkamp
in reply to james • • •Luna Lactea
in reply to Mozilla • • •Chris
in reply to Mozilla • • •Carlos Francisco 🦣
in reply to Mozilla • • •afrangry
in reply to Mozilla • • •Tofu Musubi
in reply to Mozilla • • •Jcrabapple
in reply to Tofu Musubi • • •Janie Karma S 🏳️🌈🖖🏿
in reply to Mozilla • • •Krishna Draws✏️
in reply to Mozilla • • •Ben Zanin
in reply to Mozilla • • •Kent Ahrens
in reply to Mozilla • • •Ferrous
in reply to Mozilla • • •Verwechslungsgefährte 🍿
in reply to Mozilla • • •My photo collection search could use such a thing.
#digikam
Joe Cooper 💾
in reply to Mozilla • • •Thomas Dorr
in reply to Mozilla • • •Frank Heijkamp
in reply to Mozilla • • •Hannah Kolbeck 🏳️⚧️
Unknown parent • • •@paper @sasha92 I want to say explicitly that I don't fully agree w/ @hipsterelectron here, but I think that your dismissal of their concerns also substantially misses the mark
I've also talked at length about fine details that assist the Blind folks @weirdwriter talks about in using AI above while not incurring discussed harm. Making it easy for a screen reader user to knowingly get an AI description is relatively simple and worthwhile, but giving that tool to writers needs great care
Kitchen Priestess (She/Her)
in reply to Mozilla • • •100rabh™
in reply to Mozilla • • •me_valentijn
in reply to Mozilla • • •These modules need to be opt-in from the beginning ... not mandatory with a "maybe we'll let you get rid of them some day, teehee" that seems to be your current plan.
I REALLY don't want that functionality, and especially not the bloat or other resource use. Guess it's time to start looking for a new browser.
Lambda
in reply to Mozilla • • •kira5w
in reply to Mozilla • • •Dsens
in reply to Mozilla • • •That other guy
in reply to Mozilla • • •raging insomniac
in reply to Mozilla • • •rubyda
in reply to Mozilla • • •