Repository navigation
url delims. worthwhile? #3270
Description
It would be good to maybe discuss this, as it's a common complaint.
On the one hand, the various RFCs are quite clear that whitespace, <>, and quotes are delimiters, and thus cannot ever appear in URLs. In url.js, there's this:
// RFC 2396: characters reserved for delimiting URLs.
delims = ['<', '>', '"', '`', ' ', '\r', '\n', '\t'],
Any delims that are found are thus assumed to be not part of the URL.
However, in practice, this is a bit of a deviation from the way that URLs are handled by browsers in the address bar, the window.location object, and the anchor tag, which serve as models for node's URL parser.
Would anyone object if delims were just auto-escaped?
The change in behavior would be:
url.parse("<http://foo.com>")
Currently this is parsed as
{ protocol: 'http:',
slashes: true,
host: 'foo.com',
hostname: 'foo.com',
href: 'http://foo.com/',
pathname: '/',
path: '/' }
With the proposed change, it'd be:
{ pathname: '%3Chttp://foo.com%3E',
path: '%3Chttp://foo.com%3E',
href: '%3Chttp://foo.com%3E' }
Which, incidentally, is exactly what you get if you type <http://foo.com> into a web browser address bar, or click on an <a href="<http://foo.com>">foo</a>. Yes, it is arguably not the user intent, but it's how it's interpreted everywhere else.
While this could potentially reduce the usefulness of the url module as a validator, it'd reduce a common gotcha for newcomers, and would make it a closer model to the browser, which was always the original intent of the module.
If we're going to do it, we should do it completely. We've had these little debates about escaping or not escaping spaces, and quotes, and various things one by one. Maybe we ought to just get rid of the delims altogether.
Thoughts?
/cc @bnoordhuis @3rdeden
Activity
polotek commentedon May 15, 2012
What is this common gotcha? How does it go?
indutny commentedon May 15, 2012
-1 for this, though I don't mind if it'll be hidden under some flag or option.
guiprav commentedon May 15, 2012
Could both behaviors be triggered by a flag or something? Then the question boils down to what should be the default...
isaacs commentedon May 15, 2012
@polotek The common gotcha is "Why doesn't
url.parse("http://foo.com/bar baz")work?"@indutny @n2liquid No, let's just pick one and go with it. url.parse already takes 2 more flags than I'd prefer.
indutny commentedon May 15, 2012
@isaacs -1 anyway.
s3u commentedon May 15, 2012
Actually, the string passed in the example is not a valid URL per http://tools.ietf.org/html/rfc3986 and treatment of delims should ideally be done in context as in "parse URIs from this blob of text". But AFAIU the semantics of
url.parse, no such context is expected to be assumed.isaacs commentedon May 15, 2012
@indutny Do you ever actually have delims in your urls that you need to truncate the url rather than be escaped? Without looking, what do you think this should do?
url.resolve("http://foo.com/bar", "baz<baf")?guiprav commentedon May 15, 2012
@s3u I couldn't have put better; I thought about that too but couldn't find the words. However I do believe url.parse() should assume a single-URL string context... which means this stuff should be escaped, otherwise it's not a single URL, and what does it even mean to put two URL's in there?
isaacs commentedon May 15, 2012
@s3u RFC 3986 is about URIs, and well outside the scope of node's url parser, which is only concerned with Uniform Resource Locators, of the sort used to address resources on the web.
Also, literally every single place where you can feed a URL to a thing that takes URLs, which is anywhere related to HTTP or JavaScript (ie, web browser address bar,
<a>tags, curl, and thelocationobjects in the dom) auto-escape delims rather than truncating.Delims are only relevant when parsing urls in a blob of text. If you know that the entire string represents a URL, then it seems rather common to assume that they'll be escaped.
The question is really: will this break any existing programs? What gotchas will it cause?
If the answer to both questions is "none that exist, but some that I can imagine, if I try really hard", then that's a +1 for the feature.
s3u commentedon May 15, 2012
@isaacs minor nitpick - RFC 3986 obsoleted RFC1738 - the latter specified URLs.
indutny commentedon May 15, 2012
@isaacs you convinced me
s3u commentedon May 15, 2012
Since the API already assumes a context, escaping makes sense. But docs need to be clear about the context and how delims are escaped.
kevinohara80 commentedon May 15, 2012
@isaacs none that exist, but some that I can imagine, if I try really hard +1
bnoordhuis commentedon May 15, 2012
A downside is that whitespace now becomes significant. The strings
'http://foo.com'and' http://foo.com 'currently parse to the same URL object. That will no longer be the case.guiprav commentedon May 15, 2012
@bnoordhuis & all: in the particular case of empty characters, I believe the browsers just trim the string instead of escaping.
guiprav commentedon May 15, 2012
Non-blank characters are still escaped, obviously.
kevinohara80 commentedon May 15, 2012
I do see the issue with the "trimmable" whitespace as @bnoordhuis mentions. This might be where most (if any) see issues.
isaacs commentedon May 15, 2012
Alright, seems like we have a rough consensus:
polotek commentedon May 15, 2012
I'm fine with this. Good talk.
0x80 commentedon May 15, 2012
+1!
2012.05.28, Version 0.7.9 (unstable)