Matma Rex

Welcome to Wikisource

Hello, Matma Rex, and welcome to Wikisource! Thank you for joining the project. I hope you like the place and decide to stay. Here are a few good links for newcomers:

• Help pages, especially for proofreading
• Help:Beginner's guide to Wikisource
• Style guide
• Inclusion policy
• Wikisource:For Wikipedians

You may be interested in participating in

• Proofread of the Month
• Monthly Challenge
• Requested texts

Add the code {{active projects}}, {{PotM}} or {{Collaboration/MC}} to your page for current Wikisource projects.

You can put a brief description of your interests on your user page and contributions to another Wikimedia project, such as Wikipedia and Commons.

Have questions? Then please ask them at either

• Wikisource:Scriptorium; or
• Wikisource:Scriptorium/Help

I hope you enjoy contributing to Wikisource, the library that is free for everyone to use! In discussions, please "sign" your comments using four tildes (~~~~); this will automatically produce your username if you're logged in (or IP address if you are not) and the date. If you need help, ask me on my talk page, or ask your question here (click edit) and place {{helpme}} before your question.

Again, welcome! — billinghurst sDrewth 02:49, 16 February 2018 (UTC)

Section numbers

Elsewhere you mentioned you were involved with the talk pages project, I think?

I was doing some cleanup and Gadgetifying a script that does section editing (closing discussions), and noticed it does section editing by finding the hrefon the "edit" link in the section header of the rendered HTML, and extracting the value of the "section" parameter. Which, because "Eww! Gross!", sent me down the rabbit hole looking for a sane way to do it. So far I'm coming up empty (except the 2005 bug T3605 which is for a magic word for use in templates). But I do find that the reply tool (I'm assuming) is adding a mw-comment data attribute with a JSON blob containing things like headingLevel and an id, and I would imagine that it needs to be able to edit a specific section, by number, via the API.

I was kinda surprised that section headings do not already contain its section number in an attribute somewhere, in a form that can be fed to mw:API:Edit's section parameter, because it seems a no-brainer that the flip side of allowing it in the API is making it possible to get the value from the rendered page. The alternative I can come up with is to "reverse engineer" it by implementing the reverse of MW algorithm to generate it: count the headings and assign them numbers based on their index. But while that seems relatively unlikely to change any time soon, it's a bad design and fragile in the face of future changes.

In any case… I'm hoping you either know of an existing way to do this (cleanly); or know how the reply tool solved this; or know of an alternate approach, maybe using the id the reply tool adds; or can point me at someone else that might know.

I'm thinking there's a Phab request for section numbers in rendered HTML in this at some point, but 1) I'd like to actually understand all the angles first and 2) I have a suspicion it might be a lot harder to solve than it at first seems based on the history in T3605.

PS. If you happen to know why all the attributes and ids are attached to a child span, rather than the h2/h3/etc. element itself, I'd be very curious to know the answer. It makes for needlessly gnarly code in client JS trying to deal with the stuff.

PPS. According to your global user page you're involved in Visual Editor? Ignoring for a moment that VE doesn't actually work with Extension:Proofread Page yet, I'm curious what interfaces VE provides for user scripts and Gadgets to extend it. enWS had a lot of old such tools that broke when the 2007 editor and mw.toolbar died (which is one reason some of our community members are side-loading mw-toolbar), and we're currently kinda just making do with the 2010 editor. But we'd probably benefit a lot from both the 2017 editor and VE, iff it supported Proofread Page and decent expansion options for scripts and gadgets. What is the extension story there? Xover (talk) 11:18, 29 January 2022 (UTC)

@Xover Hi, yeah, that's me. I've worked on both the talk pages project and the visual editor.

… it does section editing by finding the hrefon the "edit" link in the section header of the rendered HTML, and extracting the value of the "section" parameter …

This is, in fact, the sanest currently available way of determining the section number you need for editing. The visual editor, and the wikitext editor on the mobile site, both do it that way. Our code for this can be found around here: https://github.com/wikimedia/mediawiki-extensions-VisualEditor/blob/23b8ec2f54e24c7f03b3f2429116cd2edc827802/modules/ve-mw/preinit/ve.init.mw.DesktopArticleTarget.init.js#L190. It's not pretty, but I think it's the best thing to do right now.

Beware that section numbers may not be actual numbers, but also values like "T-1", when a section is transcluded from a different page (e.g. on pages like w:Wikipedia:Articles for deletion/Log/2022 January 24). This is to allow referring to sections inside <noinclude> or <includeonly> tags consistently. You can still use those values with the action=edit API, so it shouldn't be problematic, just make sure you don't try to convert them to integers or something.

I was kinda surprised that section headings do not already contain its section number in an attribute somewhere…

The current parser's HTML output doesn't have much useful metadata. This will be improved when we switch to Parsoid – if you look at its HTML output (e.g. https://en.wikisource.org/api/rest_v1/page/html/User_talk:Matma_Rex), you'll find nice tags like <section data-mw-section-id="1">, and the format is documented here: mw:Specs/HTML/2.4.0#Headings and Sections.

Our sites using Parsoid for normal page views is still quite a ways away, but it's actually being worked on now (after being a vague dream for many years), and it looks like it's planned for talk pages first (T272987).

But I do find that the reply tool (I'm assuming) is adding a mw-comment data attribute with a JSON blob containing things like headingLevel and an id, and I would imagine that it needs to be able to edit a specific section, by number, via the API.

Yes, those attributes are added for the reply tool. But it doesn't actually use section editing when replying; it only sends the ID of the comment and the text of your reply to the server, and the code that inserts the reply in the right place works on the whole page rather than single sections. Some of this is documented at mw:Extension:DiscussionTools/How it works if you're curious.

If you happen to know why all the attributes and ids are attached to a child span, rather than the h2/h3/etc. element itself…

I don't know why it was done this way, but if I had to guess, I'd guess that someone wanted to make it easier to style the heading text separately from the section edit links, which are outside of that <span>.

Ignoring for a moment that VE doesn't actually work with Extension:Proofread Page yet, I'm curious what interfaces VE provides for user scripts and Gadgets to extend it.

VE actually has some integration with ProofreadPage, but it's incomplete and buggy. There is a visual editing mode for the "Page" namespace, if you enable visual editor in your beta features, or try it out e.g. here (random page): https://en.wikisource.org/wiki/Page:The_Yellow_Fairy_Book_(1894).djvu/114?veaction=edit. Apparently the page scans have become invisible, which is a new bug that wasn't there the last time I tried, but this honestly reflects the level of support we have for this mode :/

As for gadgets, theoretically the entire editor is modular and anything could be implemented as a script or gadget. There is some documentation here: mw:VisualEditor/Gadgets. Unfortunately, getting anything done is more complex than with the old wiktiext editor; some of this complexity is unavoidable because editing rich text is just more complicated, some of it is because our APIs aren't so great. But still, people have managed to build some tools using it (see examples at the bottom of that page).

[For example, customizing our toolbar is a pain (but it's definitely possible). There are some reasons for it (e.g. to support buttons like "bold" and "link" activating themselves when your cursor is placed in bolded/linked text, and various buttons appearing/disappearing when you're editing a table – none of this exists in the wikitext editor), but it still shouldn't be that inconvenient to do. Anyway, I'm just grumbling right now.]

I hope that helps, and I hope I didn't miss anything. That was a long reply :) Matma Rex (talk) 15:53, 29 January 2022 (UTC)

Thanks for taking the time for such a thorough and informative reply. Very much appreciated!

That grabbing section numbers out of href is the saneest option right now is, well, pretty insane. I'll probably file a Phab for it at some point, but I imagine it'll fall into the "wait for Parsoid" black hole so many other things have disappeared into. The basic assumption that the HTML output is an actual API for various clients (community Gadget writers being the most obvious group) being one I've waited a looong time for, but that nobody wants to touch in the old parser "because Parsoid". Which worries me because it means Parsoid gets increasingly bugwards compatible with the parser we desperately want to get away from: case in point being p-wrapping (also T253072). If we don't fix grottiness in the old parser and always add grottiness to the new one, the migration is just going to end up reinventing a grotty parser we need to migrate away from.

That the section parameter in the Edit API is documented as Section number. 0 for the top section, new for a new section. but the corresponding data is not numeric for transcluded content is… annoying. That the section "numbers" for transcluded content can apparently be non-unique when multiple transclusions are present (not verified, based on some comment in Phab somewhere) tips over into infuriating. Are we ("we, the Movement"; not pointing fingers at any entity in particular) really not able to achieve better than this in 2022? /me asked rhetorically…

I think VE-with-PRP is even more buggy than is apparent. Try making a nil change to your example page and show the diff. That the page image disappears is probably a side effect of the recent addition of OpenSeadragon ("OSD") for zoom/pan/rotate etc. of the page scans, and I imagine relatively easy to fix once one goes looking. The big problem is that VE and Parsoid make assumptions about the content model that do not hold true on Wikisource. The most obvious being that a single Page: wikipage is not a single entity, but rather three parts (header/body/footer) that should probably be MCR slots but are actually implemented by using noinclude tags as delimiters and serialising to what looks like a single wikipage. Not quite as obvious is that each such wikipage is actually inherently a fragment that won't be complete until transcluded together with other fragments in mainspace somewhere. And that can be a really complex selective transclusion happening out of order, using LST and multiple invocations of PRP's <pages …> tag. The immediate reason we can't use VE on WS in Page: is T244657. And since we can't use it it never gets tested. And since we can't test it no issues get reported. Meaning there's a pretty rude awakening in store for everyone when Parsoid tries to roll out globally. Personally, I think Wikisource should be first priority for Parsoid and VE because 1) it will shake loose a lot of assumptions, and 2) if you can make them work well for Wikisource you can make them work well on any project. But insert here my standard rant that the WMF allocates zero dedicated or ongoing resources to Proofread Page or Wikisource, despite being fundamental to the operation of ~75 of the projects. We can't even get bug-fix volunteer patches reviewed in months: T296079. Or that despite every single project relying on Commons, and redirect their users there for uploads, nobody owns the multimedia stack after the Multimedia team was effectively disbanded: T294484. It's depressing always having to beg for help from people on other teams whose responsibilities do not actually include the things we need. Like T166138. And it doesn't help that there's no community representation in key decision loops, so efforts like Codex/Vue.js and the migration from Gerrit to GitLab do not actually have the community within their scope or as one of their key constituencies. Gadget writers can't even use simple ES6 syntax nicities, but we're very concerned about in-house developers being able to transpile their code and have smooth CI, because, you know, anything less would be barbaric. All of which bears somewhat on the topic at hand because…

Ugh! mw:VisualEditor/Gadgets is gnarly! It explains none of the concepts relevant to extending VE, like what the heck is a "surface". It starts by saying I need to git clone VE in order to write an on-wiki Gadget(!). I smell OOUI all over what code examples are there, and for whatever its merits it was always vastly better used from the PHP side than as an actual client-side UI library; not to mention Mediawiki is currently busy hard-deprecating it, with the only significant holdout being… VE. And OOUI was never a sane option for use in Gadgets. Just the sheer amount of boilerplate code needed to show a simple dialog box was prohibitive (I feel like I'm writing Java, ca. 1995), and the conceptual comprehension needed to do even simple things has such cognitive overhead it is effectively prohibitive for community contributors. See e.g. User:Xover/ocrtoy-prefs.js and w:User:Xover/EasyLinks.js. For community contributors you can't base your extensibility on inheritance, mixins, factory methods, or other advanced (in the sense complicated and abstract) stuff. It has to be at a reasonable level of abstraction: a list of events and a simple utility function to add a hook for them; an almost declarative way to "add a button to this section of the toolbar, with that icon image, that calls this function when clicked"; utility methods to manipulate the editor contents as text (not a node three or object hierarchy, or whatever the native representation is). How hard should it really be to do stuff like "Insert the Greek Sigma character at the insertion point", or "Wrap the selected text in {{sc|selection}}, or inset a empty one if there's no selection", or "Do this regex replace on the text in order to intelligently remove hard line breaks inside paragraphs but leave the paragraph breaks alone", or "replace all the text in the editor with this new text which I got by tweaking the OCR parameters". I'm not meaning to dump all over VE (or its docs; we all know how fun writing docs is), but I'm saying community contributors are not professional programmers (and when they are their day job is not MediaWiki or VE), are here primarily to contribute on the content side, and what coding they do (and have time / spare cycles for) is secondary. It doesn't matter how great VE's architecture is for full-time developers deeply embedded in the Mediawiki tech stack: for this group of users it just simply does not work.

But enough ranting… Yeah, mw:VisualEditor/Gadgets was the info I'd already found and which was the reason I, in boundless optimism, hoped there was a better story here. That that's more or less where we're currently at may not make me happy, but is useful to know. Mostly in the sense that it enables me to justifiably rant about it, of course, but also in that it means I'll just have to bite the bullet and dig into it if I need to write something VE-compatible. 😀

In any case: Thanks for the info and apologies for the rant(s) (which you should feel entirely free to ignore entirely unless you happen to be interested). I learned a lot from the info and links you provided so your taking the time to respond at length is very much appreciated! Xover (talk) 11:07, 30 January 2022 (UTC)

Oh, and in case you're interested: T300465, T300467. Cheers, Xover (talk) 19:47, 30 January 2022 (UTC)

@Xover That VE-with-PRP bug is awful and also really stupid and easy to fix. I've submitted a patch.

Some of the concepts of VE are documented at mw:VisualEditor/Design/Software overview, but that page might also be too much to dump on someone who just wants to add a button that does stuff. Maybe we should have a glossary page or something like that to be a middle-ground between these two…

I'll try to respond to your other points and rants (entirely justified!) tomorrow. Cheers. Matma Rex (talk) 20:52, 30 January 2022 (UTC)

@Xover Hey, sorry it took me so long to respond.

On various things:

… nobody wants to touch in the old parser "because Parsoid". Which worries me because it means Parsoid gets increasingly bugwards compatible with the parser we desperately want to get away from …

Actually there changes are being made to both parsers to bring them closer together and to ease the transition. Mostly to Parsoid, since it is the new thing that needs to conform, but the old parser too whenever that is more practical. For example, the big Tidy changes a while ago, and more recently the markup of image thumbnails is being changed in the old parser to match what Parsoid does (T51097). But this needs to be done carefully too since the current HTML output is already de facto an API, and e.g. that change to image thumbnails in fact broke stuff.

Parsoid is already bug-for-bug compatible with the old parser (well, mostly), and I don't think we can avoid that, as it would break tons of content. I don't think that there are any plans for changes that would require updating existing articles (the existence of tasks like T253072 doesn't really mean anything). At best you might get linting for when you do weird things (as in Special:LintErrors).

… section parameter in the Edit API … Are we really not able to achieve better than this in 2022? …

Well unfortunately, the section support was invented in 2003 ;) Also, writing documentation is hard, and few find it enjoyable. I'll have a look at the task about correcting it.

Is there any chance you'd be interested in submitting a patch to Gerrit to make that documentation change? (You don't even need to use Git to do it these days, in case that was a barrier – there are web interfaces for everything, although they are slightly awkward.)

The big problem is that VE and Parsoid make assumptions about the content model that do not hold true on Wikisource. The most obvious being that a single Page: wikipage is not a single entity, but rather three parts…

I think we assume that because that's what the MediaWiki API tells us, see e.g. this API query. But I know that they're not treated as simple wikitext internally (although I don't know the details) and there are ways to make the API acknowledge that, e.g. this API query. So I think that adapting Parsoid/VisualEditor to use that different format would not be insurmountable, although it would probably be a large project and it probably won't get done as long as MediaWiki can give us simple wikitext.

I have no good response to your complaints about the state of support for ProofreadPage or Wikisource. I'm not entirely happy about what WMF focuses on, but I don't feel like I'm able to change it. I am happy though that we're working on discussion pages (it only took like 10 years of community complaining that they suck) and I'm just going to work on what I can change by myself or with our small team.

And FWIW, both the code transpiling and CI are IMO also badly neglected and decidedly unsmooth. Not as neglected as ProofreadPage, but at least comparably to the multimedia support. No idea how you got the impression that WMF invests effort in those.

On visual editor and gadgets:

(I realize I'm not addressing the core issue here, but I hope it's still helpful to respond to the individual points)

It explains none of the concepts relevant to extending VE, like what the heck is a "surface".

A surface is basically the entire editor, excluding the toolbar, but including the editable content, selections, and the various popups (themselves known as contexts, inspectors and dialogs). There may be additional surfaces within the dialogs (e.g. media caption or reference contents). A surface fragment is a bag of tools for interacting with ranges of content in the surface (e.g. converting the selected paragraph to a list item).

It starts by saying I need to git clone VE in order to write an on-wiki Gadget(!).

Well, you don't need to, but it's the most convenient way to browse the source code, and browsing the source code is the best way to learn how it does things and how to make it do more things. (Maybe it's not the most accessible thing to newcomers, but it's what we have… and it does have good code comments.)

I smell OOUI all over what code examples are there, and for whatever its merits it was always vastly better used from the PHP side than as an actual client-side UI library; not to mention Mediawiki is currently busy hard-deprecating it, with the only significant holdout being… VE. And OOUI was never a sane option for use in Gadgets. Just the sheer amount of boilerplate code needed to show a simple dialog box was prohibitive (I feel like I'm writing Java, ca. 1995)…

Fun fact, OOUI actually used to be just a nameless set of widgets written for VE, before it was separated into a real library, and before the PHP version of it was written. (I wrote most of the PHP code, it was one of my first tasks after I was hired by the WMF… make of that what you will.) They definitely share a lot of patterns.

Fun fact 2, rewriting VE without OOUI was seriously proposed when Vue was being introduced into MediaWiki. It seems that we've managed to convince everyone that it would be a gigantic waste of effort, and related to that, it seems that OOUI will remain supported but discouraged for new projects. I wonder how you feel about that?

I agree that the OOUI and VE code has the Java-flavored object-oriented style, I'm not a fan of it, but I got used to it. It has some advantages for debugging (it's easy to override stuff from the browser console and backtraces make sense, unlike with the modern JS style where everything is an anonymous function defined somewhere you can't access), although writing new code in that style is a bit of a chore.

the conceptual comprehension needed to do even simple things has such cognitive overhead it is effectively prohibitive for community contributors. (…) For community contributors you can't base your extensibility on inheritance, mixins, factory methods, or other advanced (in the sense complicated and abstract) stuff. It has to be at a reasonable level of abstraction: …

I think you're underselling yourself and the community. And also, I really don't want us to dumb down things for community contributors, while keeping the real stuff for the professionals in the ivory tower at the WMF.

VE exposes the low-level stuff, but it has reasonable abstractions too; it's just not all that well documented (hence the call-out at the beginning of that guide to just go read the source code), and geared towards the things that the editor already does.

I'll try to give some examples below:

a list of events and a simple utility function to add a hook for them;

I'm not sure what you mean here.

an almost declarative way to "add a button to this section of the toolbar, with that icon image, that calls this function when clicked";

Here's how the bold button is defined: (source)

ve.ui.BoldAnnotationTool = function VeUiBoldAnnotationTool() {
	ve.ui.BoldAnnotationTool.super.apply( this, arguments );
};
OO.inheritClass( ve.ui.BoldAnnotationTool, ve.ui.AnnotationTool );
ve.ui.BoldAnnotationTool.static.name = 'bold';
ve.ui.BoldAnnotationTool.static.group = 'textStyle';
ve.ui.BoldAnnotationTool.static.icon = 'bold';
ve.ui.BoldAnnotationTool.static.title =
	OO.ui.deferMsg( 'visualeditor-annotationbutton-bold-tooltip' );
ve.ui.BoldAnnotationTool.static.annotation = { name: 'textStyle/bold' };
ve.ui.BoldAnnotationTool.static.commandName = 'bold';
ve.ui.toolFactory.register( ve.ui.BoldAnnotationTool );

It is Java-flavored verbose (although if you think that's bad, then you haven't seen Java), but it looks "almost declarative" to me. It defines the section of the toolbar (group), the icon image (icon), and the function when clicked (commandName) – which refers to a command defined here, which in turn refers to an action defined here.

The reason for the indirection is that commands can also be executed by keyboard shortcuts (triggers and sequences), or by buttons elsewhere in the interface. This doesn't exist in the old wikitext editor.

It also defines when the tool on the toolbar should be shown as active – when the cursor is in bold text (the annotation line), which is another thing that doesn't exist in the old wikitext editor.

utility methods to manipulate the editor contents as text (not a node three or object hierarchy, or whatever the native representation is).

Let me just say that this is not often useful on Wikipedia, where pages are full of links and references, and as you know the editor was created with Wikipedia in mind.

That said, you can get the current selection converted to plain text using ve.init.target.getSurface().getModel().getFragment().getText().

How hard should it really be to do stuff like "Insert the Greek Sigma character at the insertion point",

ve.init.target.getSurface().getModel().getFragment().insertContent( 'Σ' )

(if you have text selected, that will replace it – add .collapseToEnd() or .collapseToStart() before the .insertContent( … ) to change the selection first if desired)

or "Wrap the selected text in {{sc|selection}},

This is moderately tricky, because selected text may be any content and not just plain text, and because our template model is complicated because of support for complex tranclusions. If you wanted to really only use plain text, then it gets much easier.

(I'll admit this took me over 30 minutes to work out all the details, so it's probably too much to ask of someone who has not worked with VE for a few years like me. But maybe we can make this one of the examples, now that it is worked out.)

I would do it like this:

var surface = ve.init.target.getSurface();
// Get the selected content from the editor
var slice = surface.model.documentModel.shallowCloneFromSelection( surface.getModel().getSelection() );
slice.data.cloneElements( true );
// Convert our data model to HTML
var wrapper = document.createElement( 'div' );
ve.dm.converter.getDomSubtreeFromModel( slice, wrapper, ve.dm.Converter.static.PARSER_MODE );
var html = wrapper.innerHTML;
// Convert the HTML to wikitext using Parsoid API
var promise = $.post( mw.util.wikiScript( 'rest' ) + '/' + mw.config.get( 'wgServerName' ) + '/v3/transform/html/to/wikitext', {
	html: html
} );
// Display progress of the API request, prevent user from changing the selection in the meantime,
// and allow cancelling if it takes too long
surface.createProgress( promise, 'Inserting template…' );
promise.then( function ( wikitext ) {
	// Replace selected content with a transclusion
	surface.getModel().getFragment().insertContent( [
		{
			type: 'mwTransclusionInline',
			attributes: {
				mw: {
					parts: [ {
						template: {
							target: {
								href: './Template:Sc',
								wt: 'sc'
							},
							params: {
								'1': {
									wt: wikitext
								}
							}
						}
					} ]
				}
			}
		},
		{ type: '/mwTransclusionInline' }
	] )
} );

(We also have some helpers to generate that big ugly object starting with mw:, but it's all Java-flavored so I'll spare you. It's in ve.dm.MWTransclusionModel and related classes.)

or inset a empty one if there's no selection",

You can check for that using ve.init.target.surface.getModel().getSelection().isCollapsed() (in our nomenclature, a collapsed selection is just the text cursor), although it's not needed in the example above, it works naturally in this case too.

or "Do this regex replace on the text in order to intelligently remove hard line breaks inside paragraphs but leave the paragraph breaks alone",

I'm not sure what you mean by "hard line breaks". Anyway, here's a silly demo of regex replacement.

var surface = ve.init.target.getSurface();
var ranges = surface.getModel().getDocument().findText( /\ban?\b/g, { noOverlaps: true } );
// Replacing content causes offsets in the whole document from that point onward
// to shift, so do it backwards so that we don't have to adjust our ranges every time.
for ( var i = ranges.length - 1; i >= 0; i-- ) {
	surface.getModel().getFragment( new ve.dm.LinearSelection( ranges[ i ] ) ).insertContent( 'the' );
}

or "replace all the text in the editor with this new text which I got by tweaking the OCR parameters".

var surface = ve.init.target.getSurface();
var wholeDocumentRange = surface.getModel().getDocument().getDocumentRange();
surface.getModel().getFragment( new ve.dm.LinearSelection( wholeDocumentRange ) ).insertContent( 'blah blah' );

I'm not meaning to dump all over VE (or its docs; we all know how fun writing docs is), but I'm saying community contributors are not professional programmers (and when they are their day job is not MediaWiki or VE), are here primarily to contribute on the content side, and what coding they do (and have time / spare cycles for) is secondary. It doesn't matter how great VE's architecture is for full-time developers deeply embedded in the Mediawiki tech stack: for this group of users it just simply does not work.

Thanks, it's a fair criticism. And I'm also not meaning to negate it by posting all these examples (although I'm hoping to get you interested in extending VE; even if that evil plan fails, they can be integrated into that documentation page). Some of the complexity is poor design, but I'll keep insisting that some of it is also unavoidable because a rich text editor is just more complex than a plain text editor. For folks who want to deal with something simpler, wikitext and its editor are not going anywhere.

Anyway, I hope you'll like at least parts of this treatise I accidentally produced here. Matma Rex (talk) 15:22, 5 February 2022 (UTC)

How you found the time and motivation to reply, not just at length but in detail even, to my long and, I am assuming, at least partially ignorant rant I'll never know; but appreciate tremendously in any case! But now I find myself writing so long in response that I'm going to have to resort to chunked uploading to keep it manageable. :-) --Xover (talk) 15:16, 6 February 2022 (UTC)

On VE and PRP

First of all, thank you so much for fixing that VE-with-PRP bug! I am pleasantly surprised that it was a relatively easy fix, and aiui one caused by forgetting to update an internal usage in the codebase. I was honestly expecting that to require nasty hacks to get around architectural assumptions. It makes me wonder whether you've done a whole lot more work on building knowledge of PRP into VE than I've picked up on. Which in turn makes me wonder whether it might be time to actually have a discussion of where we're at and how PRP+VE should look eventually. Issues I am concerned about are: the header/body/footer division; the fact that Page:-namespace wikipages are strictly speaking fragments, meant to be transcluded together into mainspace, possibly using LST; and the fact, which follows from the preceding, that we have a hard requirement on the ability to use what VE/D/Software overview#Template Encapsulation calls "sequential model" templates across page boundaries. It doesn't help, probably, that PRP still stores this as a single wikitext blob delimited by the noinclude tags instead of using something like MCR slots, but I'm more worried about conceptual-level mismatch and conflicting architectural assumptions.

I'm not entirely up to date on what PRP does with Page: content, but I think it's roughly like this: the whole page is stored as wikitext, with the header and footer wrapped in noinclude tags. PRP extracts it on edit to present the custom edit interface, overrides the diff view to hide the plumbing, and linearilizes it on save. It overrides getSelection on the textarea so that VE can get the whole wikitext (I think for the 2017 Wikitext mode), but not the other methods (cf. my ping in the Wishlist regarding this). And it does "something" magic to extract the custom "textquality" tag from the header so it can be stored in a page property, and most recently also add a change tag when the textquality changes. This should probably be changed into three MCR slots, but… In any case, what I imagine is the big problem for VE is that not only is the Page: wikipage split in three, but we have sequential model templates with the start template in the header and the ending template in the footer. For example, {{italic block/s}} and {{italic block/e}}. Then, say, ten wikipages in the Page: namespace (each corresponding to one physical page in a scanned book) get transcluded together (by way of PRP's <pages … /> extension tag) into a mainspace page corresponding to a chapter of a book. For each Page:-namespace wikipage the body section gets transcluded but, obviously, the header and footer are noinclude'ed and get left out. It is also common to use LST to change the order of or selectively transclude parts of Page: wikipages (I have some doozies of examples to illustrate if you want them). It should end up with a consistent markup tree, but depending on where one looks it is not guaranteed (cannot be assumed) at any given step of the process.

For the "sequential model" templates I am worried that overlapping the <noinclude> tags is going to be problematic, since it's essentially the same situation as <b>foo<i>bar</b>baz</i>. And even if it's not there are definitely times when the template providing the HTML end tag is not going to be present on the page at all (either because it hasn't been added yet, or because the user simply forgot it altogether). My immediate thought is that perhaps this could be solved by VE adding a "virtual end tag" node in some heuristically chosen position, that can either be confirmed by the user, made obsolete by the user adding the real end tag in a different position, or just simply be persisted. A missing, misnested, or incorrect end tag is usually found by visual inspection on transclusion, and can sometimes be very subtle; getting a hint about it earlier in the workflow might make such issues both rarer and easier to find.

In any case, I am not necessarily convinced we can make VE (or any other WYSIWYG editor) work well for our purposes (we do too much fiddly direct formatting to reproduce the original work, which may be too inconsistent to be a good match), but I do know we rather desperately need more powerful and user friendly tools both for current and new contributors so I really hope we can at least get to a point where we can try it out.

PS. Oh, and I should be clear… I'm mentioning this stuff in case it's useful or interesting. There's no implied expectation that you'll do anything in particular in relation to it (or even reply; feel free to ignore what you're not interested in, or respond at your leisure, with no worry I'll be offended). --Xover (talk) 15:55, 6 February 2022 (UTC)

@Xover We've done some work on "building knowledge of PRP into VE" (or rather, the team has before I joined), but really not that much. VE recognizes headers and footers by looking for the markup corresponding to noinclude tags, and… that's pretty much it. There are definitely problems with "conceptual-level mismatch and conflicting architectural assumptions". VE is not aware that pages are intended to be transcluded together, and it's not aware that markup in the page body is often intended to be rendered together with other pages, and so start and end tags may be missing.

Overlapping the noinclude tags is not problematic by itself, because <noinclude> and </noinclude> are not really treated like a pair with the opening and closing tags enclosing some content, but rather as each being a separate tag. Therefore they don't require the enclosed content to balanced tags, unlike regular HTML tags. In this regard they behave much like <section begin=… /> and <section end=… /> tags, which you're probably familiar with; the wikitext markup is just misleading. Wikitext tags may look like HTML (or XML) but they're not.

However, when we try to infer the header and footer based on those noinclude tags, that is problematic. The header and footer in VE's model do require the content to be balanced, so for some markup we can't display them nicely. Here's one example: Page:Popular Science Monthly Volume 25.djvu/828 (this one uses the "italic block" template you linked), and another without many templates: Page:SpirellaCorsets page19.gif.

I agree VE may not be the best tool for Wikisource. I feel like you'd benefit from something that can edit the whole book rather than single pages, and lets you switch between previewing in paged and non-paged mode. But I haven't done any work here as an editor so I'm not sure what I'd even want myself. Matma Rex (talk) 23:44, 13 February 2022 (UTC)

Well, if we're talking pie-in-the-sky…

We spend a lot of time downloading .jp2 scan images from IA only to compile them into DjVu files because we need the book (or magazine or…) to be one atomic unit that we can point the Index: page at (currently relies on filename matching; Index:foo.pdf for File:foo.pdf). We also spend a lot of time on manipulating those DjVu files because we discover missing pages, illegible pages, out of order pages, pages with bad quality OCR, etc. If Commons could have something akin to a Category (in terms of how it looks to the end user, not necessarily in implementation) that was explicitly a "book"—with an ordered list of image files corresponding to physical pages in a paged media, and an extra MCR slot on each image for the OCR text layer—we could forget about DjVu altogether, and we could let any experienced contributor manipulate our "books" instead of relying on the extremely few hyper-technical contributors capable of writing their own image-to-DjVu-with-OCR toolchain.

We then have a pain point caused by having a couple of hundred Page:-namespace wikipages (Page:foo.pdf/1, Page:foo.pdf/2, etc.) that are subservient to the Index:-namespace wikipage, but cannot be treated as a single unit. So renaming any book here involves some form of non-core automation, or workarounds like creating a dummy Page:-namespace page (Page:foo.pdf) so we can move that, check "Move subpages", and then delete the dummy page (which also means we can easily delegate this to non-admins). Building specific knowledge of this into core (I don't offhand think an extension can cover it; but maybe with some generalised facilities in core and some WS-specific stuff in an extension) would let a lot of operations happen at the level of a whole book.

And when we get to transcluding, the top level page typically contains a little more manual input, but all the subpages (typically corresponding to a chapter) end up essentially only containing a transclusion (using PRP's <pages … /> tag, and possibly also its support for LST). For example The Science of Religion/Chapter 3, or all the subpages of In Other Words. Doing this manually using the primitive MW facilities is extremely powerful and we can usually find a way to solve any weird edge-case, but it's also tedious, labour-intensive, newcomer hostile, and error prone. Having specific knowledge of this and an "book transclusion editor" UI would be a massive improvement.

And getting briefly back on point… An editor component that has knowledge of the whole "book" as a unit, even if it still operates on individual Page: wikipages for editing, could much more readily preview-in-context, provide smooth navigation between pages (no whole page reload). I suspect current VE's role there is as an "editing widget" within the wider "book editor" rather than being the "book editor" itself. But if MW core, possibly supplemented by the Wikisource and Proofread Page extensions, supported this it's somewhere in the vicinity of realistic for VE to operate in that environment. For example because it can rely on core to provide the relationships and demarcations of units, and it could more sensibly take into account page boundary-crossing markup. It would still need to understand header/body/footer well (and, as mentioned, I think the wikitext-with-noinclude storage format needs to be replaced for that to be sanely implementable), and have specific knowledge of this, but it'd at least be a reasonable scope and separation of concerns.

It's also possible that "the editor" for this would be the 2017 wikitext editor rather than VE as such. But we have people clinging to the 2007 editor and hacky side-loading of the toolbar (some for good reasons, some less so), and I see the problems this causes so I'd like to bring us over to a more modern and supported component. Even if we only use the wikitext mode. I think there's potential for the full VE to work well for Wikisource, but it may require tradeoffs that don't make sense or be so expensive to develop and maintain that it's off the table in practice. But it's also not a conversation, I don't think, that can be sensibly had until we complete the Parsoid and VE migration (whatever definition one uses for the latter). Xover (talk) 09:51, 15 February 2022 (UTC)

On Parser-vs-Parsoid

You've guessed correctly that it's p-wrapping in Parsoid I am referring to. I am exaggerating for rhetorical purposes, of course, but it's a fact that for some things the existence of an intended future replacement Y makes it effectively impossible to get a given problem addressed in current component X. And for p-wrapping and other such bugs, it means we first need to wait for Parser to be completely removed, all the immediate problems with Parsoid fixed, and then hope someone can be convinced to tackle stuff like p-wrapping. And while p-wrapping may seem like a relatively minor and obscure issue, it was at one point close to being the straw that broke the camel's back and lost us an extremely prolific contributor (one of those people that can be turned loose on Special:LintErrors and eliminate whole classes of issues through sheer persistence and willingness to slog through cleanup tasks).

It's not a problem that gets noticed as such by end users most of the time, but anyone trying to make formatting templates (which is most of them, on Wikisource) will tear their hair out over it eventually; and end users will sooner or later run into some situation where some seemingly straightforward thing will be apparently impossible to do. Just forcing p-wrapping is bad enough because it is rigid, limits what can be done, and is outside the control of the template writer or end user. But the fact that its behaviour is also inconsistent is utterly infuriating: it makes Mediawiki feel like black magic that keeps changing and pulling the rug out from under you. Not that p-wrapping is the only thing that has that effect—templates and ParserFunctions are shockingly badly suited for the current need, to name but one—but it's my go to example for where the Parser's behaviour is clearly bad and where Parsoid has had to adopt bugwards compatible behaviour (I'm sure there are other and better examples), and it happened to be the one that nearly precipitated a ragequit situation a while back here.

I'll spare you my extended rant on this, but suffice to say I think we're at the point where either sufficient investment to finish the transition in a reasonable time needs to be made, or sufficient operating expenditure needs to be allocated to maintain both through the extended transition. [extended rant literally snipped out :)] --Xover (talk) 07:37, 7 February 2022 (UTC)

I don't know what are the plans around p-wrapping, but my guess would be that it's never going away because it'd break too much content on various projects. I am just guessing though.

As far as I can tell, all problems it causes are easily avoided by removing line breaks from your wikitext, and from any templates you use in your wikitext. This may be somewhat annoying, but surely not as annoying as the problem it solves. I remember talking to some Wikisource people on IRC who were angry about p-wrapping, and who were willing to try anything to get rid of spurious <p> tags that appeared in their books, except the one thing that reliably solves the problem, which is removing line breaks; that one they stubbornly refused to try. I genuinely don't understand why it's such a problem.

(You can write anything in wikitext with zero line breaks, although you might need to use HTML syntax for paragraphs, lists and tables. In my opinion doing that often makes it more readable, when the thing you're writing is complicated enough.) Matma Rex (talk) 00:02, 14 February 2022 (UTC)

(Also, if I may rant myself – I really hate the formatting templates used on Wikisource that cause so much trouble. When I'm asked to figure out why such-and-such page displays incorrectly, and I look at the source and it's an incomprehensible maze of brackets, pipes, and templates with two-letter names, it makes me want to run away from the whole project as far as possible. What is so wrong with <span style="…">!?…) Matma Rex (talk) 00:07, 14 February 2022 (UTC)

I like cscott's idea of a magic word that disables it for a given unit of wikitext (a template, say). For normal people editing an article on enWP they'll never see the problem; but anyone that actually runs into it would kill to be able to disable it for their use case. The biggest problem with it is how inconsistent it is, both directly because it's effectively a regex-based hack and in its effective behaviour when it interacts with our use cases. For example, we have lots of templates that take multiple paragraphs of text as an argument. Depending on whether the opening div tag of the template is on its own line or not the output will get or not get a p tag wrapped around the first paragraph, but the other paragraphs always gets a p-wrapper. And because newlines are significant in lots of contexts in MW, absent p-wrapping most of our templates would be written without newlines around the opening div.

It's interesting that you mention removing newlines as a solution. A lot of our templates contain tons of line noise in the form of HTML comments—<!-- … -->—specifically to give us something approaching code structure without introducing significant newlines. Newlines are significant in all sorts of places in MW, and Wikisource tickles most of them. Like table and list markup, that has to start on a new line; and, yes, p-wrapping that changes the behaviour depending on the presence or absence of the newline. I am also surprised that any coder would suggest "just remove all the whitespace" as a solution. Maybe we should invent a WikiScript and "transpile" it into obfuscatedminified wikimarkup off-wiki? 😎 But joking aside, we need whitespace and orderly source code formatting in both templates and wikimarkup for much the same reasons a programmer needs it in source code. The specifics vary, but the principle is the same.

There are lots of problems with the templates on enWS. Ignoring for the moment that templates (and ParserFunctions) are broken by design and completely unsuited to the role they've been employed in; and to make matters worse we're trying to do things with both templates and wikimarkup that they really haven't been designed to handle. Our collection of templates has grown organically over time, with a few distinct clusters and periods, corresponding to individual contributor's years active or areas of interest. We've also had to hack around browser limitations and compensating for an inability to get traction for our needs from Those Who Control The Resources at the WMF. At no point have we had a mechanism to modernise or standardise this into a cohesive whole. So we have some templates that take multiple paragraphs of text in argument 1; some that have /s and /e variants; some that have /begin and /end; some that have a /continue to be used instead of /s on intermediate pages. We also haven't migrated all our templates to TemplateStyles, much less Lua, because we have a much smaller technical contributor pool to pull on. We have overlapping templates like {{center block}} and {{block center}}. We have a continuous fight with the software to enable sidenotes in a sane way (and various over-enthusiastic contributors who don't know their own limitations and refuses to accept Mediawiki's). We have horrors like {{ts}} because they are, despite all appearances, the least insane way to do what we need. Don't even get me started on hackarounds to the lack of support for proper dot leaders in web standards: example.

But your reference to "just remove linebreaks" and ending rant makes me think that there's either something I don't understand about your point, something you don't understand about how WS works / its needs, or, typically, both. Could you give a few examples and some details? We have a lot of scope for improvement in that area for all sorts of reasons, but direct technical advantage like you seems to suggest has not been among the obvious benefits (which is one of the reasons I've not prioritised tackling that myself). If there's stuff we can do to reduce the friction I'm definitely very interested to hear about it.

PS. Dogfooding is often useful. I think you'd find proofreading a work on Wikisource educational. If you, or anyone else for that matter, would like to try such a project I'd be only too happy to facilitate. Not that I imagine anyone would have the time for that (neither on WMF's dime or in their volunteer time), but the offer's there should anyone be so inclined. It also wouldn't be a bad way for the WMF to build up its competence on its own projects, but I guess I'll have to track down someone with a more fancier job title to bother about that. :) Xover (talk) 10:45, 15 February 2022 (UTC)

@Xover:

I am also surprised that any coder would suggest "just remove all the whitespace" as a solution. Maybe we should invent a WikiScript and "transpile" it into obfuscatedminified wikimarkup off-wiki? 😎 But joking aside, we need whitespace and orderly source code formatting in both templates and wikimarkup for much the same reasons a programmer needs it in source code. The specifics vary, but the principle is the same.

I'm not saying that it's a good solution, but in the absence of good solutions, this one works.

Your WikiScript idea has already been invented and it's called Lua modules ;) – and if you had a look at the output generated by something like, say, w:Module:Infobox (try running some examples through w:Special:ExpandTemplates), you wouldn't object to calling it obfuscated. And yet it works!

But your reference to "just remove linebreaks" and ending rant makes me think that there's either something I don't understand about your point, something you don't understand about how WS works / its needs, or, typically, both. Could you give a few examples and some details?

I likely don't understand things, and I don't really have a point. I am also probably mixing up some vague knowledge I have of Polish and English Wikisources. I just find the syntax used for templates like Template:Ts or pl:Szablon:F unreadable, and would rather use HTML/CSS every time. And I find that simple templates like Template:Block center would not become that unreadable without line breaks, and (if I understand the p-wrapping issues correctly) it might make your lives easier.

I think you'd find proofreading a work on Wikisource educational.

So, I gave it a try. I don't want to start a new work from scratch, so I clicked through some beginner guides looking for a place to start, and I found Wikisource:Proofread of the Month, which pointed me to Index:A history of Japanese colour-prints by Woldemar von Seidlitz.djvu.

As instructed there, I looked at the discussion page to look for conventions I should follow, and immediately ran into the thing I dislike the most: apparently I must use the {{Ruby}} template to mark up Japanese names sprinkled throughout the book, instead of the normal HTML <ruby> markup. And the template doesn't even allow the text to be reproduced accurately. Honestly, not my kind of dog food.

Maybe I'll do a few pages and see how much I get yelled at. Matma Rex (talk) 23:15, 15 February 2022 (UTC)

Ouch, you sure know how to make things tough for yourself! The Community Collaboration is about works too large or too complex for individual contributors to deal with. For newcomers I recommend the Wikisource:Monthly Challenge, which is more themed works and offers several challenge levels (newbie to medium big and complex). Oh, and it's not actually a competition, despite the name; it's a community drive. Xover (talk) 06:44, 25 March 2022 (UTC)

• Matma Rex: Re {{ruby}}: as the guy who set up the template here on enWS and set up that index, don’t use the template if either you don’t want to or can’t because you need more detailed use of <⁠ruby>. The template ruby is optimized for the use of ruby formatting on Wikipedia, from which I quite shamelessly stole the template; it presupposes many things about the ruby tag, which are not necessarily true for their uses on Wikisource. It was a shorthand I used to create a simple example of ruby (ironically in English); I didn’t need any advanced technique. If you are interested in adding the ruby text for the whole of von Seidlitz, please go ahead; I would greatly appreciate the work. TE(æ)A,ea. (talk) 16:43, 25 March 2022 (UTC)

Pointers...

Hey. I haven't forgot about our sprawling multiply-threaded conversation above. It's just that IRL is sucking up every spare cycle I have just now.

But I figured you might conceivably be interested in T304303 (which springs from meta:Wikisource Triage meetings, in which I was not involved (see "IRL" above)). I'm not terribly familiar with the code in question (I know it exists, and that's about it), so I've no idea if the plan makes sense (I definitely don't agree with everything that was discussed in that meeting), but I'm sure your perspective and input would be appreciated if you want to chime in. No obligation to do so is implied, of course, just a "FYI" and an invitation iff you're interested. Xover (talk) 06:38, 25 March 2022 (UTC)

@Xover Thanks for the note. It doesn't exactly make me happy, but I think it's fair. I don't have anything useful to add. Matma Rex (talk) 07:41, 25 March 2022 (UTC)