22

✏️ Editing: lwpcook.3pm

.\" Automatically generated by Pod::Man 4.11 (Pod::Simple 3.35)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
.    ds C`
.    ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{\
.    if \nF \{\
.        de IX
.        tm Index:\\$1\t\\n%\t"\\$2"
..
.        if !\nF==2 \{\
.            nr % 0
.            nr F 2
.        \}
.    \}
.\}
.rr rF
.\" ========================================================================
.\"
.IX Title "lwpcook 3"
.TH lwpcook 3 "2024-01-25" "perl v5.26.3" "User Contributed Perl Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
lwpcook \- The libwww\-perl cookbook
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
This document contain some examples that show typical usage of the
libwww-perl library.  You should consult the documentation for the
individual modules for more detail.
.PP
All examples should be runnable programs. You can, in most cases, test
the code sections by piping the program text directly to perl.
.SH "GET"
.IX Header "GET"
It is very easy to use this library to just fetch documents from the
net.  The LWP::Simple module provides the \fBget()\fR function that return
the document specified by its \s-1URL\s0 argument:
.PP
.Vb 2
\&  use LWP::Simple;
\&  $doc = get \*(Aqhttp://search.cpan.org/dist/libwww\-perl/\*(Aq;
.Ve
.PP
or, as a perl one-liner using the \fBgetprint()\fR function:
.PP
.Vb 1
\&  perl \-MLWP::Simple \-e \*(Aqgetprint "http://search.cpan.org/dist/libwww\-perl/"\*(Aq
.Ve
.PP
or, how about fetching the latest perl by running this command:
.PP
.Vb 3
\&  perl \-MLWP::Simple \-e \*(Aq
\&    getstore "ftp://ftp.sunet.se/pub/lang/perl/CPAN/src/latest.tar.gz",
\&             "perl.tar.gz"\*(Aq
.Ve
.PP
You will probably first want to find a \s-1CPAN\s0 site closer to you by
running something like the following command:
.PP
.Vb 1
\&  perl \-MLWP::Simple \-e \*(Aqgetprint "http://www.cpan.org/SITES.html"\*(Aq
.Ve
.PP
Enough of this simple stuff!  The \s-1LWP\s0 object oriented interface gives
you more control over the request sent to the server.  Using this
interface you have full control over headers sent and how you want to
handle the response returned.
.PP
.Vb 4
\&  use LWP::UserAgent;
\&  $ua = LWP::UserAgent\->new;
\&  $ua\->agent("$0/0.1 " . $ua\->agent);
\&  # $ua\->agent("Mozilla/8.0") # pretend we are very capable browser
\&
\&  $req = HTTP::Request\->new(
\&     GET => \*(Aqhttp://search.cpan.org/dist/libwww\-perl/\*(Aq);
\&  $req\->header(\*(AqAccept\*(Aq => \*(Aqtext/html\*(Aq);
\&
\&  # send request
\&  $res = $ua\->request($req);
\&
\&  # check the outcome
\&  if ($res\->is_success) {
\&     print $res\->decoded_content;
\&  }
\&  else {
\&     print "Error: " . $res\->status_line . "\en";
\&  }
.Ve
.PP
The lwp-request program (alias \s-1GET\s0) that is distributed with the
library can also be used to fetch documents from \s-1WWW\s0 servers.
.SH "HEAD"
.IX Header "HEAD"
If you just want to check if a document is present (i.e. the \s-1URL\s0 is
valid) try to run code that looks like this:
.PP
.Vb 1
\&  use LWP::Simple;
\&
\&  if (head($url)) {
\&     # ok document exists
\&  }
.Ve
.PP
The \fBhead()\fR function really returns a list of meta-information about
the document.  The first three values of the list returned are the
document type, the size of the document, and the age of the document.
.PP
More control over the request or access to all header values returned
require that you use the object oriented interface described for \s-1GET\s0
above.  Just s/GET/HEAD/g.
.SH "POST"
.IX Header "POST"
There is no simple procedural interface for posting data to a \s-1WWW\s0 server.  You
must use the object oriented interface for this. The most common \s-1POST\s0
operation is to access a \s-1WWW\s0 form application:
.PP
.Vb 2
\&  use LWP::UserAgent;
\&  $ua = LWP::UserAgent\->new;
\&
\&  my $req = HTTP::Request\->new(
\&      POST => \*(Aqhttps://rt.cpan.org/Public/Dist/Display.html\*(Aq);
\&  $req\->content_type(\*(Aqapplication/x\-www\-form\-urlencoded\*(Aq);
\&  $req\->content(\*(AqStatus=Active&Name=libwww\-perl\*(Aq);
\&
\&  my $res = $ua\->request($req);
\&  print $res\->as_string;
.Ve
.PP
Lazy people use the HTTP::Request::Common module to set up a suitable
\&\s-1POST\s0 request message (it handles all the escaping issues) and has a
suitable default for the content_type:
.PP
.Vb 3
\&  use HTTP::Request::Common qw(POST);
\&  use LWP::UserAgent;
\&  $ua = LWP::UserAgent\->new;
\&
\&  my $req = POST \*(Aqhttps://rt.cpan.org/Public/Dist/Display.html\*(Aq,
\&                [ Status => \*(AqActive\*(Aq, Name => \*(Aqlibwww\-perl\*(Aq ];
\&
\&  print $ua\->request($req)\->as_string;
.Ve
.PP
The lwp-request program (alias \s-1POST\s0) that is distributed with the
library can also be used for posting data.
.SH "PROXIES"
.IX Header "PROXIES"
Some sites use proxies to go through fire wall machines, or just as
cache in order to improve performance.  Proxies can also be used for
accessing resources through protocols not supported directly (or
supported badly :\-) by the libwww-perl library.
.PP
You should initialize your proxy setting before you start sending
requests:
.PP
.Vb 7
\&  use LWP::UserAgent;
\&  $ua = LWP::UserAgent\->new;
\&  $ua\->env_proxy; # initialize from environment variables
\&  # or
\&  $ua\->proxy(ftp  => \*(Aqhttp://proxy.myorg.com\*(Aq);
\&  $ua\->proxy(wais => \*(Aqhttp://proxy.myorg.com\*(Aq);
\&  $ua\->no_proxy(qw(no se fi));
\&
\&  my $req = HTTP::Request\->new(GET => \*(Aqwais://xxx.com/\*(Aq);
\&  print $ua\->request($req)\->as_string;
.Ve
.PP
The LWP::Simple interface will call \fBenv_proxy()\fR for you automatically.
Applications that use the \f(CW$ua\fR\->\fBenv_proxy()\fR method will normally not
use the \f(CW$ua\fR\->\fBproxy()\fR and \f(CW$ua\fR\->\fBno_proxy()\fR methods.
.PP
Some proxies also require that you send it a username/password in
order to let requests through.  You should be able to add the
required header, with something like this:
.PP
.Vb 1
\& use LWP::UserAgent;
\&
\& $ua = LWP::UserAgent\->new;
\& $ua\->proxy([\*(Aqhttp\*(Aq, \*(Aqftp\*(Aq] => \*(Aqhttp://username:password@proxy.myorg.com\*(Aq);
\&
\& $req = HTTP::Request\->new(\*(AqGET\*(Aq,"http://www.perl.com");
\&
\& $res = $ua\->request($req);
\& print $res\->decoded_content if $res\->is_success;
.Ve
.PP
Replace \f(CW\*(C`proxy.myorg.com\*(C'\fR, \f(CW\*(C`username\*(C'\fR and
\&\f(CW\*(C`password\*(C'\fR with something suitable for your site.
.SH "ACCESS TO PROTECTED DOCUMENTS"
.IX Header "ACCESS TO PROTECTED DOCUMENTS"
Documents protected by basic authorization can easily be accessed
like this:
.PP
.Vb 5
\&  use LWP::UserAgent;
\&  $ua = LWP::UserAgent\->new;
\&  $req = HTTP::Request\->new(GET => \*(Aqhttp://www.linpro.no/secret/\*(Aq);
\&  $req\->authorization_basic(\*(Aqaas\*(Aq, \*(Aqmypassword\*(Aq);
\&  print $ua\->request($req)\->as_string;
.Ve
.PP
The other alternative is to provide a subclass of \fILWP::UserAgent\fR that
overrides the \fBget_basic_credentials()\fR method. Study the \fIlwp-request\fR
program for an example of this.
.SH "COOKIES"
.IX Header "COOKIES"
Some sites like to play games with cookies.  By default \s-1LWP\s0 ignores
cookies provided by the servers it visits.  \s-1LWP\s0 will collect cookies
and respond to cookie requests if you set up a cookie jar. \s-1LWP\s0 doesn't
provide a cookie jar itself, but if you install HTTP::CookieJar::LWP,
it can be used like this:
.PP
.Vb 2
\&  use LWP::UserAgent;
\&  use HTTP::CookieJar::LWP;
\&
\&  $ua = LWP::UserAgent\->new(
\&      cookie_jar => HTTP::CookieJar::LWP\->new,
\&  );
\&
\&  # and then send requests just as you used to do
\&  $res = $ua\->request(HTTP::Request\->new(GET => "http://no.yahoo.com/"));
\&  print $res\->status_line, "\en";
.Ve
.SH "HTTPS"
.IX Header "HTTPS"
URLs with https scheme are accessed in exactly the same way as with
http scheme, provided that an \s-1SSL\s0 interface module for \s-1LWP\s0 has been
properly installed (see the \fI\s-1README.SSL\s0\fR file found in the
libwww-perl distribution for more details).  If no \s-1SSL\s0 interface is
installed for \s-1LWP\s0 to use, then you will get \*(L"501 Protocol scheme
\&'https' is not supported\*(R" errors when accessing such URLs.
.PP
Here's an example of fetching and printing a \s-1WWW\s0 page using \s-1SSL:\s0
.PP
.Vb 1
\&  use LWP::UserAgent;
\&
\&  my $ua = LWP::UserAgent\->new;
\&  my $req = HTTP::Request\->new(GET => \*(Aqhttps://www.helsinki.fi/\*(Aq);
\&  my $res = $ua\->request($req);
\&  if ($res\->is_success) {
\&      print $res\->as_string;
\&  }
\&  else {
\&      print "Failed: ", $res\->status_line, "\en";
\&  }
.Ve
.SH "MIRRORING"
.IX Header "MIRRORING"
If you want to mirror documents from a \s-1WWW\s0 server, then try to run
code similar to this at regular intervals:
.PP
.Vb 1
\&  use LWP::Simple;
\&
\&  %mirrors = (
\&     \*(Aqhttp://www.sn.no/\*(Aq                       => \*(Aqsn.html\*(Aq,
\&     \*(Aqhttp://www.perl.com/\*(Aq                    => \*(Aqperl.html\*(Aq,
\&     \*(Aqhttp://search.cpan.org/distlibwww\-perl/\*(Aq => \*(Aqlwp.html\*(Aq,
\&     \*(Aqgopher://gopher.sn.no/\*(Aq                  => \*(Aqgopher.html\*(Aq,
\&  );
\&
\&  while (($url, $localfile) = each(%mirrors)) {
\&     mirror($url, $localfile);
\&  }
.Ve
.PP
Or, as a perl one-liner:
.PP
.Vb 1
\&  perl \-MLWP::Simple \-e \*(Aqmirror("http://www.perl.com/", "perl.html")\*(Aq;
.Ve
.PP
The document will not be transferred unless it has been updated.
.SH "LARGE DOCUMENTS"
.IX Header "LARGE DOCUMENTS"
If the document you want to fetch is too large to be kept in memory,
then you have two alternatives.  You can instruct the library to write
the document content to a file (second \f(CW$ua\fR\->\fBrequest()\fR argument is a file
name):
.PP
.Vb 2
\&  use LWP::UserAgent;
\&  $ua = LWP::UserAgent\->new;
\&
\&  my $req = HTTP::Request\->new(GET =>
\&      \*(Aqhttp://www.cpan.org/CPAN/authors/id/O/OA/OALDERS/libwww\-perl\-6.26.tar.gz\*(Aq);
\&  $res = $ua\->request($req, "libwww\-perl.tar.gz");
\&  if ($res\->is_success) {
\&     print "ok\en";
\&  }
\&  else {
\&     print $res\->status_line, "\en";
\&  }
.Ve
.PP
Or you can process the document as it arrives (second \f(CW$ua\fR\->\fBrequest()\fR
argument is a code reference):
.PP
.Vb 3
\&  use LWP::UserAgent;
\&  $ua = LWP::UserAgent\->new;
\&  $URL = \*(Aqftp://ftp.isc.org/pub/rfc/rfc\-index.txt\*(Aq;
\&
\&  my $expected_length;
\&  my $bytes_received = 0;
\&  my $res =
\&     $ua\->request(HTTP::Request\->new(GET => $URL),
\&               sub {
\&                   my($chunk, $res) = @_;
\&                   $bytes_received += length($chunk);
\&                   unless (defined $expected_length) {
\&                      $expected_length = $res\->content_length || 0;
\&                   }
\&                   if ($expected_length) {
\&                        printf STDERR "%d%% \- ",
\&                                  100 * $bytes_received / $expected_length;
\&                   }
\&                   print STDERR "$bytes_received bytes received\en";
\&
\&                   # XXX Should really do something with the chunk itself
\&                   # print $chunk;
\&               });
\&   print $res\->status_line, "\en";
.Ve
.SH "COPYRIGHT"
.IX Header "COPYRIGHT"
Copyright 1996\-2001, Gisle Aas
.PP
This library is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

MR SKB