Posts in GroupServer Development
Between testing, and moving to a new server, it has taken quite a while to ship
the new sign up code. However, it is going now on all our sites. I also have
some acceptance-tests that I can post, once I remove all the client-specific
tests from it!
So I do not forget (or a least, so GroupServer remembers) "htmllaundry" is a module for sanitising HTML markup, based on the "Cleaner" class in lxml. (Get it? html *laundry*, cleans code and removes *lint*. Oh my aching sides…). I especially like the support for Zope 3 Schemas, so fields that expect HTML input are explicitly marked.<http://pypi.python.org/pypi/htmllaundry/1.2>
I was playing around with HTML 5 video and audio elements. Specifically, I was looking at how hard it would be to add video and audio to posts. From the coding-side it looks like it will be easy to provide support for them, but there is one rather large catch when it comes to video encoding formats. HTML is the language used to describe the content of web pages. The language has been very stable since last version of the standard, HTML 4, which was released over ten years ago. However, work is almost complete on the new version, called HTML 5. One of the new features of HTML 5 are elements that describe video and audio content, in much the same way as the "img" element describes images. Work is already under-way implementing the video and audio elements in all new web browsers, except for Microsoft Internet Explorer. GroupServer could add video and audio tags whenever someone posted a video or audio as an attachment. The results would look like the screenshots "video1.png" and "audio.png" below. The HMTL code itself is very small — no more than what we do for the existing thumbnail images, and is a lot less that the code required to embed Flash video <http://dev.opera.com/articles/view/introduction-html5-video/>. There is one fly in the ointment: not all audio and video formats are supported. Currently Mozilla Firefox and Opera support the free Ogg Theora for video <http://www.theora.org/> and Ogg Vorbis for audio <http://www.vorbis.com/>. Google Chrome supports Ogg Theora and the commercial H.264 video; Apple Safari only supports H.264 video (Apple claims that the lack of hardware support is a problem for Theora, which is a bit odd as Apple is a hardware company). A video in an unsupported format (such as H.264 in Mozilla Firefox or Opera) will come up with a box much like the "video2.png" screenshot below. For more complete support the videos could be converted from H.264 to Theora or vice versa. I am unsure if software that will do the conversion requires a payment of fees to the MPEG LA. (Apple and Google has paid their fees, which is why H.264 is supported in the commercial browsers Apple Safari and Google Chrome, but not their open-source equivalents, Webkit and Chromium). Once a common video format is supported for viewing video I suspect that content-creation in that format (from cameras and their supporting software) will become common. The standards battle between Ogg Theora and MPEG LA H.264 will be an interesting one to watch from the sidelines. My guess is that Wikipedia will be the one to force Theora support into all browsers, as they only support open standards (which is also why all the audio on Wikipedia is in Ogg Vorbis format <http://en.wikipedia.org/wiki/Helter_Skelter_%28song%29>). Anyway, that is enough fun for a rainy Saturday ☺
The following files were added to this topic:
I have had an opportunity to tweak registration using the Web: • The order of the pages has changed, • The Set Password page has been simplified, • There is a new progress indicator, • The wording has been tweaked, and • There has been some reworking of the code. This is the first change to registration in almost 18 months! The order of the pages that are seen during registration has changed. The Set Password page has now moved to before the Change Profile; the new page order is now 1. Enter email 2. Set password 3. Change profile 4. Verify email This makes it very unlikely that the new member will be left without a password set. In turn, this makes it easier to get the new member back to completing his or her registration if, for some reason, registration was abandoned. I need to work on the initial sign-up page to help new members get back on track more easily — resolving an issue we have had since the registration system was designed in [Michael checks the topic] 2007. The set password page has been simplified, by changing the Password entry to show plain text, rather than a series of dots. This allows the new member to know what he or she has typed, so the Repeat Password field can be dropped. I muted the idea in 2009 September <http://groupserver.org/r/post/333ThiwHcQyPpUxf429WVo/>; for the last couple of months Alice and I have trialled it with the registration process of a big client of OnlineGroups.Net. It has worked very well, so I am shipping it as part of the main GroupServer code. As shown in the image below, registration has a new progress indicator. This will hopefully give the new member some idea of how many tasks are required before registration is complete. This idea was also trialled with the Big Client, and it is working well. However, the Big Client got a progress indicator that had a graphical progress bar. To make a progress bar for every skin would be very time consuming, so I opted for the simpler bullet-point progress indicator, which sits in the spot normally taken up by the context-navigation menu. (My argument is that the progress indicator is showing you your context, even if you cannot navigate with it.) With this new progress indicator, the current task is bold and has a triangle-bullet *‣* *Like* *this* Completed tasks are indicated by a tick-mark ✓ Like this Finally, tasks that need to be done have no indication. I have reworked the wording on the pages. For the most part it has involved adding some bold text, and saying “thanks” more. I have also tried to make the wording less formal, while still being precise and clear. Finally, and least importantly, the code has been reworked so all the registration pages are in their own module "gs.profile.signup". Previously the code was mixed up with the main profile code for GroupServer, "Products.GSProfile", which is huge. I split the code out in an attempt to preserve my sanity. A huge thanks to Alice and Dan, who have supported the trial of the new registration code, and dealt with a lot of the support load. Also thanks to the Big Client for allowing us to use your members to test out the ideas. Finally, thanks to members themselves for the excellent descriptions of the problems that occurred. Next up for me is testing, because I could have broken all sorts of things without knowing it. I hope to have the new code deployed on OnlineGroups.Net by next Tuesday.
The following file was added to this topic:
And so I do not forget: GroupServer currently relies on PIL 1.1.6.
On Tue, Dec 1, 2009 at 1:45 PM, <email obscured>> wrote: > Thanks, Jim! > > Thunderbird not understanding links is insane, but good to know. I wonder how many people do *not* have that installed? Every Thunderbird client I have seen has had URLs that can be clicked on, but I have not seen that many to be honest. Well, thinking about it, it's possible that links of the format http://something may have been handled all along, but a few hours ago I had an email with text saying just domain.com; I noticed that Google linkified them, and Thunderbird did not, until I added the plugin. So it's possible that I'm talking only about the severely degenerative case here. Yes, a test confirms ... well-formed URLs are linkified properly without the addon. False alarm. Nothing to see here. Move along!
Thanks, Jim!
Thunderbird not understanding links is insane, but good to know. I wonder how
many people do *not* have that installed? Every Thunderbird client I have seen
has had URLs that can be clicked on, but I have not seen that many to be
honest.
On Mon, Nov 30, 2009 at 4:41 PM, Michael JasonSmith <email obscured>> wrote: > * Does any modern email client not make text such as > http://groupserver.org/ click-able? Even the text-based Pine and > Elm have links you can click on thanks to GNOME Terminal > understanding them. (I would believe that Apple Mail does not > understand links, but I have a very low opinion of that client.) Thunderbird doesn't, unless you install the "text link" add-on (or possibly others).
I love the idea of making the posts on the web look better ☺ As you say,
Steve, first impressions count, and many more people will see a pretty
post on the Web than in email.
I do not see what the users would gain by creating a HTML version of a
plain-text message.
* Does any modern email client not make text such as
http://groupserver.org/ click-able? Even the text-based Pine and
Elm have links you can click on thanks to GNOME Terminal
understanding them. (I would believe that Apple Mail does not
understand links, but I have a very low opinion of that client.)
* I created a little experiment ages ago, where words are marked
up with <strong> if they are surrounded by asterisks like *this*
— and I have seen very few people use it. Off the top of my
head, no one but myself. There is little point adding a system
to interpret mark up if no one will use the markup. (I tried
bold first because I have seen italic done /in/ _many_
''ways''.)
* GroupServer could try and identify bullet lists in the body of
the email, but I do not see what using • instead of * would gain
anyone. I use the bullet character in my posts to ensure that
the UTF-8 handling is going ok, not because it looks better ☺ In
addition, recognising bullet lists in plain text is not easy, as
many irritating wiki systems will testify.
I'd be most interested in something that simply allowed embedded links, bold, italics and perhaps bullets with the font, size, and color kept uniform. While hiding links to spoof sites is a problem with spam, we do our best to keep spammers and scammers off of our groups. I too think presenting HTML on the web posts is actually more important than in e-mails. Anything to make the groups look more blog-like is good. First impressions count. Steven Clift - http://stevenclift.com Executive Director - http://E-Democracy.Org Follow me - http://twitter.com/democracy
On Mon, Nov 30, 2009 at 1:12 PM, <email obscured>> wrote:
> The problem of HTML mail is two-fold: formatting and security.
Well, the HTML format email that Steven was talking about was to
increase the richness of communication from the site to the user; not
to preserve the formatting on the original post.
So you could look at it from the point of view that "all incoming
posts are plaintext" and "outgoing emails may be formatted"
The problem of HTML mail is two-fold: formatting and security. First is the issue of formatting the message. What should we do if a message has attached images referenced from the HTML? Should we strip the images? Would the message be readable without the bright purple image in the background? Should we sanitise the HTML so it is vaguely conforming to a known standard? The other problem is security. The "style" attribute, for example, is a know vector for JavaScript attacks. For security we could strip the style attributes, and other known problems, but then the messages may not look good. We could leave the known vectors in, and claim we are just a messenger, but I suspect that would not go down very well with many members. By allowing HTML links in a message the author could claim, in the link text, that he or she is linking a financial report but instead send the other members to YouTube < This could also be used to send users to sites that will infect their machines with malware. (On Slashdot they put the domain-name after every link, to prevent this from happening <http://tech.slashdot.org/story/09/11/29/2133243/G-WAN-Another-Free-Web-Server?art_pos=2>.) I am more interested in experimenting with displaying HTML-formatted messages on the Web pages. This would allow us to deal with the issues in a known medium, and then move onto the trickier one ☺
Thanks for your ideas, Steve. Group members typically receive and read every message, as most groups are small and the members are very interested in what is going on. Because of this I doubt that digests will be the normal way of receiving notification about activity in a group for most people in most groups. However, in large active groups, such as the ones on the e-Democracy.org Forums, the picture is quite different. Many members are interested in what is going on, but not so interested that they want to read every message. For example, in the Minnesota Issues Forum <http://forums.e-democracy.org/groups/mpls/> • 35% of members receive a digest, and • 28% of members receive no notification. (These numbers will be a little high, as we do not remove the delivery setting when a person leaves the group, but they give the general idea.) I would be keen to see an HTML format of the digests sent out along with a text-version. At the same time, I would love to see the information scent of each entry in the digest increase. By increasing the information scent we should help the members find topics that they want to read and comment on. For example, the Web version of each topic lists the files that have been added to the topic and the keywords in the topic. The digest messages do not list either of these things. Combining digests from multiple groups is an interesting idea, albeit a painful one to implement. Currently GroupServer goes through all groups that require a digest to be sent, constructs a digest, and sends the digest out to the members that want to receive it. Constructing a unique message for every member — comparing the digests that he or she wants to receive against the list of available digests — would be a far slower operation. For the record, close to 200 members of the e-Democracy forums receive more than one digest, compared to the 940 members that receive only one digest. Because of this, and the high complexity of sending unique digests, I would favour tackling the HTML-formatted digests first.
A rough idea I'd like to throw into the mix. Replace separate digests for those on multiple groups with a single customized site-wide HTML default formatted topic digest covering ALL the forums they are a member. This can be a reliable place to track what is new and would allow site hosts to also include various organizational messaging of the day/week/etc. to communicate strategically with participants. I can see a time emerging where the digest mode becomes the default delivery option for groups with e-mail requestable for all posts and by topics of greatest interest. Steven Clift - http://stevenclift.com Executive Director - http://E-Democracy.Org Follow me - http://twitter.com/democracy
What are folks current thinking about an HTML option for posts these days? I am coming around to the position that embedded links in the text would be a good thing now that HTML e-mail is pretty much universal. It could also be use to present links to attached files placed on the server and thumbnails of photos in a box of sorts in the top right of a message. I've noticed with my DoWire posts that if via Gmail I send a rich text formated e-mail (typically) to a posting address that GroupServer (I believe OR is something else going on) converts the links into this format: http://groups.dowire.org/groups/newswire/messages/post/4NiSNuISbZxW90ru0HbfXf What are you folks thinking? Steve Steven Clift - http://stevenclift.com Executive Director - http://E-Democracy.Org Follow me - http://twitter.com/democracy
I love it when small changes make things easier for people. Steven Clift - http://stevenclift.com Executive Director - http://E-Democracy.Org Follow me - http://twitter.com/democracy On Mon, Nov 23, 2009 at 9:12 PM, <email obscured>> wrote: > I have changed GroupServer so it provides feedback when a user submits a form. Now, when user clicks the submit button, the button will become disabled (greyed out) and its text is changed to “Processing…”. > > Besides providing feedback to the user, disabling the submission button should prevent problems caused by the user double-clicking on the button, which can submit the form twice. (A problem I entirely blame on Steve Jobs, but that is a rant for another day and another medium.) > > The process of the disabling the button is a little more complex than I wanted. We use the zope.formlib library for creating and processing forms. It relies on the identifier of the button being sent from the browser as part of the form information. However, disabling the button means that its identifier is omitted from the for data. To get around this I hide the button and put a new (disabled) button in its place. Browsers happily send the ID of a hidden button to Zope, so everything works fine. > > Thanks to Richard who helped me solve this nasty little problem. > > Dan, One day I will fix the paste-issue with required widgets in GroupServer forms.
I have changed GroupServer so it provides feedback when a user submits a form.
Now, when user clicks the submit button, the button will become disabled
(greyed out) and its text is changed to “Processing…”.
Besides providing feedback to the user, disabling the submission button should
prevent problems caused by the user double-clicking on the button, which can
submit the form twice. (A problem I entirely blame on Steve Jobs, but that is a
rant for another day and another medium.)
The process of the disabling the button is a little more complex than I wanted.
We use the zope.formlib library for creating and processing forms. It relies on
the identifier of the button being sent from the browser as part of the form
information. However, disabling the button means that its identifier is
omitted from the for data. To get around this I hide the button and put a new
(disabled) button in its place. Browsers happily send the ID of a hidden button
to Zope, so everything works fine.
Thanks to Richard who helped me solve this nasty little problem.
Dan, One day I will fix the paste-issue with required widgets in GroupServer
forms.
I mused about converting the PDF documents HTML. However, I have never
been happy with the PDF documents that Google has automatically
converted to HTML: the resulting HTML does not read like either a Web
page or a PDF.
I suspect that Google originally converted the PDFs to HTML in order to
index them, for much the same reason as you propose converting them,
Richard ☺
On Fri, 2009-11-06 at 11:30 +1300, Jim Cheetham wrote:
> I think it's a great idea overall, and helps to remove the problems
> caused when one member of a group "upgrades" to the latest MS Office
> version and starts distributing files that the other members cannot
> read. I'm not a fan of PDF being any form of long-term storage option,
> but for these purposes it sounds very handy.
At the expense of me offering to triple our storage requirements, we
could convert to HTML at the same time, for a really 'quick read'. We
effectively have to do a plain text conversion for indexing anyway.
On Fri, Nov 6, 2009 at 10:54 AM, <email obscured>> wrote:
> Richard, you did not misunderstand me; I communicated poorly. Sorry, Jim.
No, I think we're all talking about the same thing. If an ODF comes
in, you cannot generate thumbnails directly; you need to create a PDF
of it, and to generate thumbnails of that.
This meant two things to me; one was the possibility that you could
make that PDF available for people that cannot read ODF natively
(hence the extra link, and the concern that the UI might make readers
think the original writer had made the PDF)
The second was terminology-related; you were talking about the PDFs
being thumbnails, which I had probably misinterpreted :-)
I think it's a great idea overall, and helps to remove the problems
caused when one member of a group "upgrades" to the latest MS Office
version and starts distributing files that the other members cannot
read. I'm not a fan of PDF being any form of long-term storage option,
but for these purposes it sounds very handy.