Making a release of GroupServer 14.03

Hi Bill,

Thanks for agreeing to help out with the release of GroupServer
14.03. Below I will outline what the goals are for GS 14.03, and
how I make a release.

The primary goal of GS 14.03 is to release the new UI. (The
install process does need to be attacked with a machete and
glyphosate, but I am going to leave that to GS 14.05.) The
secondary goal is to update the dependencies. I normally make a
GroupServer update in two stages corresponding to those two
goals, fixing any problems as I go.

*Stage* *one*

1. Grab the buildout-code¹.

2. Update the versions.cfg and buildout.cfg so they have an
up-to-date list of packages and versions. This normally means
sitting down with Meld² and merging what is in buildout with
what is on Production and on my development box. I have the
relevant sections of the various cfg files below.

3. Decide what to do with the shim. I suggest chucking it: it
can only cause issues.

4. Create a tar-ball
$ hg archive -ttgz -X".hg*" -pgroupserver-1403a \
/tmp/groupserver-1403a.tar.gz

5. Unpack the tar-ball somewhere and see if it builds. As you
well know, this takes a while, so it is a good chance to work
on other things.

* If there are issues then fix the problems and go to step 4.
* Otherwise go to Stage two.

If you manage to get even this far I will be pleased!

*Stage* *two*

1. Unpack the new GS tar-ball (which is known to work) from
Stage one.

2. Set the index to PyPI, rather than the GS cache:
index = http://pypi.python.org/simple

3. Set the "allow-hosts" to the following:
eggs.iopen.net
pypi.python.org
d.pypi.python.org
zope.org
www.zope.org
svn.zope.org
sourceforge.net
wwwsearch.sourceforge.net
initd.org
github.com
cloud.github.com
plone.org
cheeseshop.python.org
fabfile.org
nongnu.org
savannah.nongnu.org
www.amk.ca
www.pycrypto.org
pycrypto.sourceforge.net
www.lemburg.com
www.egenix.com
zif.svn.sourceforge.net
download.zope.org
svn.sqlalchemy.org
code.google.com
formencode.org
formencode.sf.net
svn.formencode.org
dist.plone.org
www.ijg.org
pytz.sourceforge.net
prdownloads.sourceforge.net
www.stuartbishop.net
undefined.org
pylockfile.googlecode.com
svn.plone.org
github-media-downloads.s3.amazonaws.com
bitbucket.org
packages.python.org
wiki.zope.org
zope2.zope.org
cvs.zope.org
docutils.sourceforge.net
codespeak.net
lxml.de
www.nongnu.org
ftp.dlitz.net
feedparser.googlecode.com
svn.red-bean.com
svn.pythonmac.org
www.replortlab.com
launchpad.net
www.simplistix.co.uk
pybrary.net
stompstompstomp.com
smontanaro.dyndns.org
www.sqlalchemy.org
downloads.egenix.com
download.zope.org
ag-projects.com
www.reportlab.com
smontanaro.dyndns.org

4. Grab the latest Zope versions.cfg and ztk-versions.cfg
configuration files from the latest version of Zope 2³.

5. Update the configuration files in the new GS directory.

6. Try and build.

* If there are version-conflict errors try and fix them, and
repeat.

7. Once all the errors have been sorted
* Update the configuration in buildout,
* Make a new tar-ball,
* Unpack it, and try again with a new install.
* If there are errors go to 6.

8. Post the new tar-ball. I will then use it to update the cache
we maintain of all the packages that GroupServer relies
on. Then we'll open up the build to others to try.

*Footnotes*

1. The buildout code
<https://source.iopen.net/groupserver/buildout/>

2. Meld <http://meldmerge.org/>

3. The Downloads page at Zope 2
<http://download.zope.org/Zope2/index/>

production-versions.cfg 9.69kb

mpj17-versions.cfg 9.84kb

mpj17-buildout.cfg 5.29kb

production-buildout.cfg 5.33kb

Hi Michael,

Well, stage one is complete. Attached is a tarball that, with the guidance at
http://pages.e-democracy.org/Installing_GroupServer_and_E-Democracy (which I
only refer to because of the bit about having to edit the postgres config), I'm
able to install and start the spiffy new GroupServer on a vanilla Ubuntu VM.

In a day or two I'll try to get through stage two.

groupserver-1403a.tar.gz 53.33kb

Thanks, Bill.

I am glad that stage one progressed without too much trouble. I hope you
have good fortune with the second stage!

Stage 2 Updates, with good news and bad news.

First, the need to edit postgres configuration kept annoying me. I know it's a
pretty simple edit, but I'd like to think in a perfect world (which we're
always striving for right?) a user setting up a basic GroupServer instance
shouldn't have to edit postgres files. So, I've modified gs_install_ubuntu.sh
so it will automatically edit the appropriate postgres configuration file if
required. It's been tested and works. In addition, I made sure that all the
table/user creation commands are explicitly being executed on the port(s)
specified in config.cfg (in retrospect, the samething should happen for
hostname too).

As far as updating Zope: the good news is there were no version conflicts (save
for Zope again wanting to have setuptools = 0.6c11, which I switched to
setuptools >= 0.6c11, since any buildout package from the last couple of years
has moved beyond setuptools 0.6). GroupServer builds and starts with Zope2
2.13.21 and ZTK 1.0.8. Buildout files for this are attached.

The bad news is a set of runtime errors I encountered.

First, if one starts GroupServer as a foreground process (./bin/instance fg)
and then exits via Ctrl-C, the attached build will throw an exception caused by
ZODB trying to close database connections via a method that no longer exists.
Traceback at http://pastebin.com/2yASSt8g . This does not appear to happen when
stopping a GroupServer deamon (./bin/instance stop).

Second, encoding. So far I've only seen it come up once, but I suspect it might
come up a lot more. On GroupServer's homepage, the attempt to fetch a list of
topics via ajax causes an exception when Products.GSSearch attempts to pass
keywords through unicode(). Traceback at http://pastebin.com/e9GhH6iz

Finally, I suspect this has less to do with version changes and more to do with
me still not understanding what hostnames are suppose to be set to what. This
time, I followed the instructions at http://groupserver.org/downloads/install/
and tried an installation that did not use a web proxy. I was able to access
GroupServer at <hostname>:8080, but all of the generated resource requests
(images, CSS, etc...) where to <hostname>:80, making for one big old plain page
of text.

14.03-stage2.tar.gz 59.73kb

I suppose that Unicode error means
https://source.iopen.net/groupserver/gs.core/summary is relevant? I always find
the decode/encode terminology used in the Python would backwords, so I'm not
sure if to_ascii is needed, or if it's to_unicode_or_bust.

First, *please* leave all installation issues until after this release. I
know that the entire thing is dodgy, and tricky, and hard, and
annoying. However, it is more important that we make a release *at* *all*
than improve the installation process.

Second, great news about the lack of version conflicts! Thank you so much!
I will have a detailed look tomorrow.

As for the shutting down, you are not alone, and it seems like a new thing:
<http://webcache.googleusercontent.com/search?q=cache:hcA8qbI_wa0J:https://mail.zope.org/pipermail/zodb-dev/2013-August/015119.html>
I am unsure what to do about this.

I agree that gs.core needs to be busted out to resolve the Unicode
issue. In this case gs.core.to_unicode_or_bust. I will make the change
tomorrow morning.

*Aside:* Sane strings are in Unicode. They are *encoded* to a sequence
of bytes (using the ASCII, or UTF-8 encodings). A sequence of bytes (in
either ASCII or UTF-8) is *decoded* into a sane Unicode string.

The big ol' page of text is odd: all the resource-requests are from "/",
with no port at all, I thought. Could you check the URLs please, Bill?

Sorry for the exuberance on postgres. You're right that we should focus on one
thing at a time.

The issue with resource requests seems to be that the <base> tag is being set
to //HOSTNAME. I'm looking at the source of the home page served by a
GroupServer instance set to hostname=gstest; port = 8080. The base tag has
href="//gstest". The requests themselves are relative URLs.

One last bit: having gone through the installation probably half a dozen times
over the last few days, I've never been able to visit a local GroupServer
instance unless both host and zope_host are set the hostname of choice in
config.ini.

I totally forgot that GS can run on a port other than 80. I will try and fix
that soon.

I have made a fix for the Unicode issue in GSSearch:
<https://source.iopen.net/groupserver/Products.GSSearch/changeset/6d79f16338879f5c444b42ae04ebd864dd6da066>

Maybe we should have just one config-item for the zope-and-gs host.

Hi dear people,

I'm very much interested in deploying GroupServer into the services
architecture at http://allmende.io . I cannot see mailman anymore.

When I just registered here, it wouldn't allow me to put an Umlaut in the
password.
Is that intended?

Well I'm very much looking forward to the 14.03 release.
Please let me know if I can be release-testing for you.

This request is very urgent.

Kind regards,

jon

Below are the release notes for GroupServer 14.04 (Ozuo utilised as an
unguent). If anyone could find the time to read them over and comment on them
it would be appreciated :)

release-notes-1404.txt 18.63kb

release-notes-1404.html 35.45kb

Hey Michael,

I read through and made a few proof-reading changes to the attached text
version. The only other points of feedback I have are:

* I think it's worth mentioning which parts of Bootstrap are and are not
included with GroupServer.

* The section "The GroupServer Installation documentation" does not exist in
this document, but it is linked to.

release-notes-1404.txt 18.58kb

Thanks for the changes, Bill.

I have updated the release notes for Ouzo¹. I have also updated the
versions.cfg file with what is in my development platform². Next up is testing
everything in its current state, fixing the <base> element to include the port,
and finally seeing if the host-name issue can be mitigated.

*Footnotes*

1.
<https://source.iopen.net/groupserver/buildout/changeset/2a42c8221cd570de61db45a2b5d3e3db50512475>
2.
<https://source.iopen.net/groupserver/buildout/changeset/16f548ba3a6da340b2cb07898359c5cafc8c5c5d>

Hi Bill,

Could you please check my change to fix the issue with the missing port?
<https://source.iopen.net/groupserver/gs.content.layout/changeset/461888711ceb10cca657534f4e888afbf7860053>

I have pushed up the change as
gs.content.layout = 2.0-20140319174723-461888711ceb

Hi Bill,

Could you please test the tar-ball of GroupServer 14.03 below? If anyone else
wants to test please feel free to try it, too!

I get a new site with the tar-ball below. However, I am not relying on our
cache for the eggs (yet) so I usually get a timeout (or two) during the build.
I also have not tested everything with the site. Ok, all I have tested is "does
the homepage render" and "does the Group page render". There is a lot to test.

groupserver-1403g.tar.gz 59.29kb

Hi Michael,

I successfully installed GroupServer 14.03a on a vanilla Ubuntu VM, started
GroupServer, visited it, and posted to the Example Group.

Huzza

Thanks, Bill!

In groupserver-1403.tar.gz below is what I am optimistically thinking of as the
“release candidate” for GroupServer 14.03 (Ouzo utilised as an unguent). It
differs from the version that Bill tested only in where it gets the eggs from
(just our servers, to reduce problems with the network timeouts that tend to
plague PyPI and other sources). As shown in the screenshots below I can also
start a topic, post, and change a profile image.

Could others please test the tar-ball below? Jon, this is the one for you ☺
Richard? Alice? Is it possible for either of you to set GS installing in the
background?

I will update the version numbers in the installation documentation. The
installation process will remain the same.

groupserver-1403h.tar.gz 58.85kb

gs-1403.png 51.50kb

The posted image gs-1403-example_group.png

gs-1403-example_group.png 105.74kb

I did find an error in GS 14.03h, but I fixed it. The tar-ball remains
unchanged. Quite how this all works out will require some explanation, which I
provide below.

One of the new packages in Ouzo is premailer, which is used to add the CSS to
the HTML-formatted notifications. The current version of premailer is 1.3.0
(according to PyPI <https://pypi.python.org/pypi/premailer/> What I think
happened is that version 1.1.3 of premailer was inadvertently called 1.13. When
buildout downloads eggs it installs the newest by default; 1.13 > 1.3 (in
version-number math) so 1.13 was installed. This broke the notifications.

I fixed the issue by downloading the tar-ball of premailer 1.3.0 from PyPI,
uploading that to our egg-repository, and deleting premailer version 1.13. So I
fixed the error, but the tar-ball remains unchanged.

Hi Michael,

I've gone over to the Mac dark side for the most part but I'll try to test it
out on an Ubuntu box this weekend.

On 20 Mar 2014, at 18:27, Michael JasonSmith <<email obscured>> wrote:

> ― Attachment links are at the end of this email ―
>
> Thanks, Bill!
>
> In groupserver-1403.tar.gz below is what I am optimistically thinking of as
the “release candidate” for GroupServer 14.03 (Ouzo utilised as an unguent). It
differs from the version that Bill tested only in where it gets the eggs from
(just our servers, to reduce problems with the network timeouts that tend to
plague PyPI and other sources). As shown in the screenshots below I can also
start a topic, post, and change a profile image.
>
> Could others please test the tar-ball below? Jon, this is the one for you ☺
Richard? Alice? Is it possible for either of you to set GS installing in the
background?
>
> I will update the version numbers in the installation documentation. The
installation process will remain the same.

Rest of post

Thanks, Alice. This weekend is fine :)

When we are about to make a new release of GroupServer, I like to see if
I can install it myself. Many experienced sysadmins and developers
report problems installing GroupServer. I am neither a sysadmin nor a
developer. In theory, I have no chance of success. But I like to try. As
expected, my attempts to install GroupServer throw many false negatives.
If I actually succeed, then we (mainly Michael) must be doing something
seriously right.

To offset my disadvantages, I use a platform as close as I can find to
the supported one. Like Alice, I now dwell in the shadows of MacOSX. I
decided to try to install GroupServer in a virtual machine.
Extraordinarily, and with the help of "The Ultimate Setup Guide: Ubuntu
13.04 in Virtualbox 4.2 on Mac OS X"
<http://www.lecloud.net/post/52224625343/the-ultimate-setup-guide-ubuntu-13-04-in-virtualbox>
I successfully installed Ubuntu 13.10 server in a virtual machine.

With help from the install documentation, and Michael, I sshed into the
ubuntu machine, edited the hosts and the config files, copied and
untarred the tarball and ran the installer. First go, the installer
stopped because my postfix smtp settings were broken and the
installation process tries to send email. This is something we are going
to look at next release. This time, with help from Michael and the god
of Accept-the-defaults, I resolved it by getting smtp to work.

The second problem to occur was an unavailable upstream package. This
was temporary as it turned out. Re-running the installer worked.
GroupServer started.

Then I faced the challenge of browsing the new GroupServer site in order
to confirm success. Pushes at doing that from MacOSX met resistance. My
Ubuntu install had no gui. Although my practical skills as a sysadmin
are barely existent, I do have one or two general notions. One of these
was that there is actually such as thing as a character-based browser.

Colleagues, I present in the attached (see link below), the output of:

dan@ubuntuVM:~$ links http://gstest:8080

Rest of post

The posted image links_groupserver_1403_group.png

links_groupserver_1403_group.png 65.81kb

The posted image links_groupserver_1403_site.png

links_groupserver_1403_site.png 109.22kb

I know this might be the only time somebody visits a GroupServer site using a
text browser, but it does make me happy to know that GroupServer looks pretty
good on a console.

FYI Alice and Dan, there have been attempts to install GroupServer on OS X:
http://groupserver.org/groups/development/messages/topic/4GbdLdhcs10UDX5tUD5unm/
. They haven't ended in success yet (last time I tried it appeared I would have
to recompile the Python interpreter), but I think it just might be possible
with enough perseverance/stubbornness.

Sorry for the delay, I had some issues installing Ubuntu 13.10 onto my old box.

I ran into a Postfix/SMTP problem that I am assuming is the same one Dan came
up against. (Postfix configuration was always the bane of installing
GroupServer for me.) For the record, I told Postfix that everything was
"local". I've attached the install output below, and also took the opportunity
of making some notes on the installation process, since it was a fresh install
and the first time I'd done that in a while.

* "Install the required packages" comes before "Download" but I had to
download GS to even see
these instructions, so the order seemed odd.
* Dependencies are listed in a table, which is nice, but a command to copy and
paste would have
saved time.
* Given that "Install the required packages" is the first instruction, I typed
out the apt-get install
command myself, then stopped and didn't run it when I saw that the install
process would install
the dependencies anyway.
* Then, after I edited config.cfg, I realised I did need to install at least
postgres, so that I could continue
with the configuration before actually running the install script. So I
ended up running the apt-get
command to install all dependencies anyway.
* Any sort of advice on what sort of postfix installation to make would be
helpful.

Install output:

alice@viola:~/groupserver-1403h$ ./gs_install_ubuntu.sh
Installing dependencies

Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
build-essential is already the newest version.
postfix is already the newest version.
postgresql is already the newest version.
python is already the newest version.
python-dev is already the newest version.
sed is already the newest version.
python-virtualenv is already the newest version.
redis-server is already the newest version.
swaks is already the newest version.
libpq-dev is already the newest version.
The following extra packages will be installed:
libxml2-dev
The following NEW packages will be installed:
libxml2-dev libxslt1-dev
0 upgraded, 2 newly installed, 0 to remove and 266 not upgraded.
Need to get 1.420 kB of archives.
After this operation, 4.815 kB of additional disk space will be used.
Get:1 http://de.archive.ubuntu.com/ubuntu/ saucy/main libxml2-dev i386
2.9.1+dfsg1-3ubuntu2 [844 kB]
Get:2 http://de.archive.ubuntu.com/ubuntu/ saucy/main libxslt1-dev i386
1.1.28-2 [576 kB]
Fetched 1.420 kB in 0s (1.535 kB/s)
Selecting previously unselected package libxml2-dev:i386.
(Reading database ... 197944 files and directories currently installed.)
Unpacking libxml2-dev:i386 (from .../libxml2-dev_2.9.1+dfsg1-3ubuntu2_i386.deb)
...
Selecting previously unselected package libxslt1-dev:i386.
Unpacking libxslt1-dev:i386 (from .../libxslt1-dev_1.1.28-2_i386.deb) ...
Processing triggers for man-db ...
Processing triggers for doc-base ...
Processing 1 added doc-base file...
Setting up libxml2-dev:i386 (2.9.1+dfsg1-3ubuntu2) ...
Setting up libxslt1-dev:i386 (1.1.28-2) ...

Testing mail setup...

<** 550 5.1.1 <<email obscured>>: Recipient address rejected: gmail.com
-> QUIT
<- 221 2.0.0 Bye

Rest of post

Thanks for giving it a try, Alice.

The "local" option for Postfix is a bit of a misnomer. What I think it means is
"accept everything for user@example-machine, and reject everything else." The
consequence is that the email addresses specified in the config.cfg have to
match up exactly with what has been set up on the local box.

I think what you want is a "satellite system" pointed at another SMTP box. If
you want to give it a push then you can re-run the configuration for Postfix:

$ sudo dpkg-reconfigure postfix

For the next release I hope to come up with some way to just *dump* the
messages to the file system by default. (This is what I first interpreted
"local" as meaning!) I am also going to drop the "groupserver user", which will
also simplify things.

I have re-written the Requirements section, so the installation-script is
mentioned first. Thanks for your suggestion, Alice.

Both the Requirements section and the Download section suffer from the
dual-nature of installation documentation. It serves as both documentation that
is shipped with GroupServer, and on the site
<http://groupserver.org/downloads/install/> On the site the Requirements acts
as a warning (for those that hate PostgreSQL look elsewhere, for example) and
the Downloads sections makes sense. In an ideal World there would be two
documents, and the time to keep them synchronised.

Oh, for what it is worth, I made the release of 14.04 (Ouzo utilised as an
unguent) today:
<http://groupserver.org/r/topic/7apc0CDT4ReT0DrKfsnQy5>

Michael, thank you.

Implementing the new UI was a huge step forward for GroupServer. The
improvements to adding people, notifications and many other small things
all add up to a more enjoyable and productive experience using
GroupServer. It is wonderful to have this available for download and use
for free under an open source licence.

My thanks also to Bill and Alice for help with packaging and testing,
and to Mike Harding and Karl von Randow of Cactuslab for the UI design.

Rest of post

Sorry I didn't get to test again before the release. Looks good though,
congratulations!

Alice

Rest of post

You cannot post because you are not logged in.

Making a release of GroupServer 14.03

Already a member?

Not a member?