Replacing Ripper by Prism

Description

It would be great to replace the usages of Ripper in IRB by Prism.
There are many advantages:

Prism is faster than Ripper, and the difference is even more visible on TruffleRuby, where irb is visibly less reactive because of Ripper.
Prism provides a much better API
Prism works well on all Ruby implementations, Ripper is a big hack which is very hard to support on all Ruby implementations (at least it has proven difficult to maintain on CRuby, JRuby and TruffleRuby, right now TruffleRuby is updating to Ruby 3.3 and Ripper is proving to be again a pain to get working, it even has a copy of st.c now to give an idea, on top of depending on many internal CRuby files).
Prism is actually used in CRuby now as the default parser since Ruby 3.4 preview 2 and is the only parser in TruffleRuby, so it is the source of truth when it comes to parsing Ruby code.

Could this be considered? What are the challenges?

@tompng

I do consider this but not prioritizing it before Ruby 3.4. Its priority will depend on when IRB becomes a bundled gem too, which will make having prism gem as its dependency easier.

@tompng what do you think?

Its priority will depend on when IRB becomes a bundled gem too

According to https://github.com/ruby/ruby/blob/90ef28f59b4a2214d64f21d7e6d033273a0cce89/lib/bundled_gems.rb#L33 in 3.5.0.

The boilerplate of Rails 8 contained irb. And the minimum support version of Rails 8 is Ruby 3.2.

If we replace ripper to prism, we need to add prism to dependency of irb for Ruby 3.2 and Rails 8. It means everyone compile prism gem for their Rails application.

I wonder if this is really beneficial to irb and Rails users.

With the transition to more bundled gems it seems Rails already has many native dependencies.
For instance:

$ ruby -v
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
$ gem i rails
...
Building native extensions. This could take a while...
Successfully installed racc-1.8.1
...
Successfully installed websocket-extensions-0.1.5
Building native extensions. This could take a while...
Successfully installed websocket-driver-0.7.6
Building native extensions. This could take a while...
...
Successfully installed rails-7.2.2
46 gems installed

Also:

$ rails new foo
$ cd foo
$ rm -rf vendor/bundle
$ bundle install | grep 'with native'
Installing stringio 3.1.1 with native extensions
Installing io-console 0.7.2 with native extensions
Installing bindex 0.8.1 with native extensions
Installing racc 1.8.1 with native extensions
Installing msgpack 1.7.3 with native extensions
Installing bigdecimal 3.1.8 with native extensions
Installing date 3.3.4 with native extensions
Installing nio4r 2.7.4 with native extensions
Installing json 2.7.5 with native extensions
Installing websocket-driver 0.7.6 with native extensions
Installing psych 5.1.2 with native extensions
Installing bootsnap 1.18.4 with native extensions
Installing debug 1.9.2 with native extensions
Installing puma 6.4.3 with native extensions

So there are already plenty of native gem dependencies for rails, and I think there is no choice with the transition to more default & bundled gems.

Ah that was rails 7.2.2, but anyway the situation is similar on rails-8.0.0.rc2:

$ gem i rails --pre
$ rails new --skip-bundle rails8
$ cd rails8
$ bundle install | grep 'with native'
Installing stringio 3.1.1 with native extensions
Installing io-console 0.7.2 with native extensions
Installing bcrypt_pbkdf 1.1.1 with native extensions
Installing bindex 0.8.1 with native extensions
Installing racc 1.8.1 with native extensions
Installing msgpack 1.7.3 with native extensions
Installing date 3.3.4 with native extensions
Installing nio4r 2.7.4 with native extensions
Installing bigdecimal 3.1.8 with native extensions
Installing ed25519 1.3.0 with native extensions
Installing json 2.7.5 with native extensions
Installing websocket-driver 0.7.6 with native extensions
Installing psych 5.1.2 with native extensions
Installing bootsnap 1.18.4 with native extensions
Installing puma 6.4.3 with native extensions
Installing debug 1.9.2 with native extensions
$ grep ' rails ' Gemfile.lock
    rails (8.0.0.rc2)
  rails (~> 8.0.0.rc2)

It means everyone compile prism gem for their Rails application.

Is it any different than other extensions like bigdecimal (which is really unavoidable for any web app realistically)?
I checked if prism took longer to install but prism and bigdecimal both take around 10 seconds to build for me locally:

gem i prism  16.41s user 1.04s system 161% cpu 10.794 total
gem i bigdecimal  8.95s user 1.31s system 103% cpu 9.946 total

I think it is a good idea to start implementing with Prism. But currently I'm not prioritizing it high. It already work in CRuby with ripper. Needs some effort to completely reimplement colorizing and nesting level calculating, and it will take time.

irb is visibly less reactive because of Ripper.

Maybe this is because IRB is calling Ripper::Lexer#scan three times per each key stroke.

it has proven difficult to maintain on CRuby, JRuby and TruffleRuby

Ripper took some work to implement initially but has been no more difficult than the parser to maintain going forward. JRuby has fully-functional Ripper support without any native dependency currently (though this may change as we move toward Prism).

With the transition to more bundled gems it seems Rails already has many native dependencies

Only on implementations that require native libraries for their extensions. On JRuby, none of the standard libraries require building at install time and they can all be installed without a C compiler present on the host system (though some, like fiddle, still need make).

no more difficult than the parser to maintain going forward.

Which is a huge effort on every Ruby update (I would say by far the biggest task for any Ruby version update in alternative Ruby implementations, obviously depends on how many lexer/parser changes there were, thankfully Prism solves this now).
Also this means JRuby would have to keep maintaining that Java lexer & parser just for Ripper, that's a big overhead.
BTW there seems to have been plenty of inconsistencies.

On JRuby, none of the standard libraries require building at install time and they can all be installed without a C compiler present on the host system (though some, like fiddle, still need make).

Prism is a standard library since 3.3 (default gem to be exact) so I guess that's something to figure out for installing Prism on JRuby (e.g. via a -java gem using WASM/Chicory or by publishing precompiled binary gems for prism like some other gems using FFI).

JRuby would have to keep maintaining that Java lexer & parser just for Ripper, that's a big overhead.

As long as CRuby ships with Ripper, we'll have to do this regardless. So it isn't going to matter much to us if Ripper or Prism are used by IRB, except that Prism will require an external native library (WASM could indeed help here).

Prism is a standard library since 3.3 (default gem to be exact) so I guess that's something to figure out

Once we start shipping with Prism (JRuby 10) we will include in the distribution a build of Prism for as many supported platforms as possible. Outside those platforms, an additional install will be required. That will either build a binary at install time, fetch a pre-built binary, or use WASM.

We will not ship without a parse.y-based parser until well after JRuby 10, and perhaps longer if we have to continue supporting Ripper.

To be fair, if everything moved to Prism and Ripper was no longer needed tomorrow, we'd be willing to drop it. But it has become quite established and it will take years for third-party libraryes to replace it. Chicken/egg... CRuby will keep shipping it because libraries use it, and libraries will keep using it because CRuby ships it.

Moving IRB to Prism from Ripper would be a positive move, but perhaps premature; Ruby 3.3 has been out for less than a year and all previous releases do not ship with Prism.

(I hit update too soon...)

Once we've had a few releases of CRuby that ship with Prism standard, I'd be more comfortable having key standard libraries like IRB depend on it.

I also concur with headius, coming from another point of view (indirectly that is). 3.3.6 was recently released (which also has issues; ruby-gtk3 does not work right now, and I use the latter as my primary test-GUI, for other GUIs e. g. ruby-libui, jruby-swing - I use the same underlying code base for all GUI-work by the way - but also for the same variant on the web, where I also use ruby consistentyl), and I had a few minor issues with ruby-dev before 3.3.6, so I think taking this slowly is better right now - let stability kick in first. Naturally I'd like to see for things becoming a bit more stable overall (I also had the rust jit fail to compile ... too many things aren't working that well right now from my point of view), so perhaps in 2025 this here could be reviewed. eregon is naturally preferring a switch, but any more balanced review should, in my opinion, also take in trade-offs users and developers may face.

Currently, these default/bundled gems uses Ripper.

irb power_assert rbs rdoc syntax_suggest

Maybe... removing ripper dependency from all of these gems can be discussed at bugs.ruby-lang.org?

In RDoc, I think there is a strong motivation to switch to Prism because of maintainability reason. RDoc's traditional parser using ripper still does not support endless method yet. RDoc already have experimental implementation using Prism.
In other gems, motivation to switch might not be high.

I'm currently trying to make an experimental implementation using Prism.

Colorization: Working on this branch. Test all passes with tweaking some tests. Symbol colorization improves.
Indent and nesting calculation: Trying and struggling now. I think it will take time to implement and to verify the quality.

In defense of the idea in general... with prism becoming the default parser in 3.4, ripper is bound to start drifting away as prism enables syntax changes that are difficult to represent in parse.y. Eventually all libraries will need to move off of ripper, but it may be years before that's feasible.

@tompng

Coming back to this, how about merging @tompng's PRs? (linked above this comment)

Hi folks - can someone more educated than me please help me understand how users are intended to produce portable java/jruby 9.x (rails) applications after this change?

In particular due to the rails dependency on irb, irb 1.17.0 seems to now break a pre-packaged/portable jruby application seemingly because the prism native extension (unlike other rails native dependencies) has no java/jruby equivalent.

Prior to this, rails 7.2.3 was able to be installed just fine without truly native extensions - the gems noted in the lists above are either optional (e.g puma, bootsnap) or had java implementations (e.g date, psych, bigdecimal, racc, io-console, nio4r, websocket-driver) .

Maybe

there's a bundler hack/workaround I am not aware of, or
perhaps it works on JRuby 10 where prism is standard
or perhaps I am doing something unexpected somewhere when packaging my app with gems that is causing this

It's possible I need to dig in further to figure out what's going on here; so just looking for a direction, as right now it's just giving Caused by: org.jruby.exceptions.StandardError: (GemNotFound) Could not find prism-1.9.0 in locally installed gems during bundler materialize upon rails boot on some platforms.

build(deps): downgrade irb to 1.16 to remove prism dependency

@chadlwilson

@chadlwilson IMO that should be reported on the JRuby issue tracker, or maybe the Prism issue tracker in addition to JRuby.
"portable java/jruby 9.x (rails) applications" is not something IRB maintainers are working on.

@eregon

@eregon ok, fair enough. I was really just looking at whether this would be expected since there was discussion above about effect of adding more native dependencies on rails applications so I infer some consideration for the backward compatibility effects on consumers of adding the dependency.

Aside from that, highlighting that this appears to be a breaking change for some user configurations who end up with irb at production time regardless of whether they want/need it (rails users, alas).

If I understand correctly, this would also mean CRuby apps potentially needing to change any mechanism for pre-bundling/caching gems across platforms if they intend to build an application to be deployed across platform/arch where prism is their only native gem.

I was really just looking at whether this would be expected since there was discussion above about effect of adding more native dependencies on rails applications so I infer some consideration for the backward compatibility effects on consumers of adding the dependency.

Yes, adding the dependency is intentional (necessary for older Ruby versions) and expected that it requires installing Prism.
On CRuby & TruffleRuby I don't think this will change much, any non-trivial application already has native gem dependencies.
On JRuby it's less common, though e.g. any gem using FFI to a library built by a gem would have the same problem too.

IOW, I think this is something JRuby devs need to solve in coordination with Prism, e.g. to use the Prism wasm backend. It's tricky though because from the POV of RubyGems if you compile an extension on one platform and run on another it's not supported because the extension is not available for that platform. Maybe having java platform prism gem would make sense then, but I don't know much about those.
For the general case (i.e. other gems using FFI) though you'd need to compile on every platform you want to run on, no way around that.

Navigation Menu

Repository navigation

Replacing Ripper by Prism #1024

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Replacing Ripper by Prism #1024

Description

Description

Activity

st0012 commented on Oct 28, 2024

eregon commented on Oct 28, 2024

hsbt commented on Oct 31, 2024

eregon commented on Oct 31, 2024

eregon commented on Oct 31, 2024

eregon commented on Oct 31, 2024

tompng commented on Oct 31, 2024

headius commented on Nov 2, 2024

eregon commented on Nov 2, 2024

headius commented on Nov 2, 2024

rubyFeedback commented on Nov 5, 2024

tompng commented on Nov 6, 2024

headius commented on Nov 6, 2024

eregon commented on Jan 8, 2026

chadlwilson commented on Feb 19, 2026

eregon commented on Feb 19, 2026

chadlwilson commented on Feb 19, 2026

eregon commented on Feb 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions