Thanks a lot Julian! Great work!

Very interesting. One of my colleagues recently did the exact same thing and ran into some issues with the Garbage collector. Basically we figured out that as long as the LotusScript Object (can't remember exactly what it's called) that enables communication with the Java code is kept alive it won't close its network connections. The Domino webserver hewas retrieving http pages from was configured for a maximum of 40 worker threads, so if he needed to retrieve 41 pages in one agent-run the code would stop and the Domino server would stop serving html.
We are definately going to have a look at what's in your box

J-P: hey, thank YOU for helping me test, and giving some good suggestions. If not, I would have been on the 1.2 release already as people came up with all the things that you did.

Jens: Wow, that's interesting. As a quick test, I wrote this agent:

Dim fetcher As New UrlFetcher
Dim url As String
Dim i As Integer

For i = 1 To 50
'** I tried with and without this line
Set fetcher = New UrlFetcher
url = "http://localhost?" & i
Call fetcher.getUrlAsString(url)
Print " Attempt #" & i & ": " & _
fetcher.GetReturnStatus & ", url = " & url
Next

I ran it against a Domino 7 server on localhost from a Notes 6.5 and a Notes 7 client, and I even ran it as a web agent ON the Domino server (just for kicks). The server was set up for 40 threads like yours was, and I didn't notice any problems.

But then again, I can see how there could be problems like the one you're describing. I'm not sure how the Java sockets work with regards to releasing connections. I know in my code for this database I try to make sure to explicitly close the connection InputStream, which I think will close the connection too.

Also, with the LS2J stuff, if you're creating a lot of objects I think you can call "Delete" against the JavaSession object to release the resources, and then create a new JavaSession. Not sure how immediately the resources are released, but that may help in some situations too.

- Julian

So by cross platform you mean any Notes/Domino platform that supports Java? Oh well, my 7.0.2 client on the Mac will have to look elsewhere to render notes mashups. Of course I'm being facetious - very nice work guys...

LOL. Yeah, we always forget about you Mac guys... Sorry 'bout that. Maybe that's a good excuse for me to finally get a Powerbook or something.

Thanks Julian! Great work (as always!).

And also good timing. Two weeks ago we wrote a small LS2J wrapper to get some HTML in a LotusScript agent...

Now your lib is WAY better and even documented

PS: Will you post it on OpenNTF, even without Bruce' help? *grin*

Hi,

I think you are not closing the OutputStreamWriter (method getUrlAsStream) properly?
writer.close() after writer.flush()?

Maybe jakarta.HTTPClient can be used from Domino7. I am going to try that. But your class allready has a lot of the features like basic authentification, ssl, proxy. I can see no support for different encoding shemes like UTF-8 or ISO-8859-1. From my experience you often have to mess with those.
The advantage of jakarta.HTTPClient is, that it allready takes care of low level stuff like closing connections and supporting features like encoding, which can be overlooked easily.

kind regards
Axel

When setting acl-wise default on no-access and a technical user on manager access, I don't get the method UrlFetcherJava.setBasicAuthentification working. I am using Domino 7
There was an error: Illegal character(s) in message header value: QWRtaW4gQXhlbDprZW5ud29ydA==

Thats because of: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4615330

I patched URLFetcherJava.createAuthHeader like that to make it work:
sun.misc.BASE64Encoder encoder = new sun.misc.BASE64Encoder();
String str = username + ":" + password;
auth = "Basic " + encoder.encode(str.getBytes());
(the last line is changed).

Axel -

Thanks (as always) for the help. I'll have to check on closing the OutputStreamWriter. Closing it early might do something bad to the connection, and I need to see if the connection might close it automatically. It certainly does look like something that could leak though, and we don't want that.

Definitely try the Jakarta classes. I've used those in other projects before (not sure about HTTP, but definitely some of the others), and they are very solid. They will definitely be more feature-rich, and I would trust those more than my own code...

Also, thanks very much for the link to the HTTP authentication bug, and your workaround code. Especially the workaround code! It's always the little things that get you. I'll make the change in the download database.

- Julian

A slight tangent, but I would suggest avoiding the sun.misc stuff, i.e. the Base64 classes -- they're undocumented, and whilst pretty true to the spec., I prefer to go with something a little more "known".

I've used the Apache commons codec library with no issues:

http://jakarta.apache.org/commons/codec

Julian,

I tried to use your code with this agent:

Sub Initialize
Dim fetcher As New UrlFetcher
Dim fileName As String

Call fetcher.AddPostParam("q", "Ronald van Gogh" )
fileName = fetcher.getUrlAsFile("http://www.google.com")

' if we successfully got the URL, say what the file name is
If (fileName "") Then
Print "URL retrieved as file: " + fileName
Else
Print "There was an error: " + fetcher.getLastError()
End If
End Sub

However it results in an error: stream closed.
Without specifying the AddPostParam it seems to work.

Any ideas?

Ronald

Ronald: I would suppose that Google simply doesn't accept POST requests. You probably need to create a query string and send a GET request.