Resolving missing YAML issue in Perl CPAN

When installing modules as part of your Perl ecosystem or when updating modules inside your Perl ecosystem you do most likely use CPAN or CPANM. CPAN stands for Comprehensive Perl Archive Network and is the default location where a large set of additional modules for the Perl programming language do reside and which you can download. CPAN provides an interactive CLI interface which you can use to install new modules. In essence CPAN is much more then just a tool and a download location, CPAN is a thriving ecosystem where people add new software and modules to the extending CPAN archive on a daily basis. The below image shows how the CPAN ecosystem looks like from a high level perspective;

When you use the CPAN CLI out of the box you might be hit with a number of warnings around YAML missing. YAML stands for, YAML Ain't Markup Language, and is a human friendly data serialization standard for all programming languages with implementatins for C/C++, Ruby, Python, Java, Perl, C#, .NET, PHP, OCaml, JavaScript, ActionScript, Haskell and a number of others.

The warning messages you get when using CPAN might look like the ones below;

"YAML' not installed, falling back to Data::Dumper and Storable to read prefs '/root/.cpan/prefs"

"Warning (usually harmless): 'YAML' not installed, will not store persistent state"

The way to resolve this is very simple, first ensure you have the YAML module installed. If this is the case ensure you inform CPAN that you would like to use YAML, this can be done by excuting the 2 commands below and this should resolve the issue of the repeating warnings when using CPAN.

o conf yaml_module YAML
o conf commit

Perl use an array variable

Perl, as many other languages has a array variable type. Accoording to Wikipedia an array is the following in computer science: In computer science, an array type is a data type that is meant to describe a collection of elements (values or variables), each selected by one or more indices (identifying keys) that can be computed at run time by the program. Such a collection is usually called an array variable, array value, or simply array. The use of an array is a very effective way of storing variabels in something that is most comparable to a list of things.

In the below example we stored some of the names of a phonetic alphabet in the variable phonetic:

my @phonetic = ( "Alpha", "Bravo", "Charlie", "Delta" );

Now if we want to do something with it we can call the variable @phonetic however this would give you the entire collection of all values in the array. For example using the print command:

my @phonetic = ( "Alpha", "Bravo", "Charlie", "Delta" );

print @phonetic;

this would give you a result you most likely do not want, namely:

AlphaBravoCharlieDelta

As you can see this is printed without any space between it or a newline. Simply using all the values from the array in the print command at once. In many cases you would like to take all the values of the array one by one. In some cases, for example adding up all the numerical values in the array, you might want to use it in this way however in most cases you would like to loop value per value.

In the example below we loop the array and do a print for every value in the array.

my @phonetic = ( "Alpha", "Bravo", "Charlie", "Delta" );

foreach (@phonetic) {
 print $_ . "\n";
}

When running this example you will see that we do not get the result all in one line as was with direct print on the array. Now we will have a result as shown below;

Alpha
Bravo
Charlie
Delta

In some cases you would like to do an action on a certain value. Every value in an array has an index number (starting at 0). So lets say we want to print the value Charlie we have to call the array value with the index number 2. The below example will print the value "Charlie"

my @phonetic = ( "Alpha", "Bravo", "Charlie", "Delta" );

print @phonetic[2];

Bloom filter stuff

Bloom filters is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set. False positives are possible, but false negatives are not. The Bloom filter was conceived by Burton H. Bloom in 1970

Where X is in Y like comparisons can be quite resource intensive, code having comparisons and equations like this can run for days when you execute them against large datasets. Bloom filters can be a big relieve and save you a lot of resources and time.

In Perl for example this is a lookup hash, a handy idiom for doing existence tests:

foreach my $e ( @things ) { $lookup{$e}++ }

sub check {
my ( $key ) = @_;
print "Found $key!" if exists( $lookup{ $key } );
}

When running this against a small set of data and in a situation where time is not a very big issue this will work fine. However if one or possibly both are against you you might want to use a bloom filter. In Perl this would look something like this:

use Bloom::Filter;

my $filter = Bloom::Filter->new( error_rate => 0.01, capacity => $SONG_COUNT );
open my $fh, "enormous_list_of_titles.txt" or die "Failed to open: $!";

while (<$fh>) {
chomp;
$filter->add( $_ );
}

sub lookup_song {
my ( $title ) = @_;
return unless $filter->check( $title );
return expensive_db_query( $title ) or undef;
}


An empty Bloom filter is a bit array of m bits, all set to 0. There must also be k different hash functions defined, each of which maps a key value to one of the m array positions.

For a good hash function with a wide output, there should be little if any correlation between different bit-fields of such a hash, so this type of hash can be used to generate multiple "different" hash functions by slicing its output into multiple bit fields. Alternatively, one can pass k different initial values (such as 0, 1, ..., k-1) to a hash function that takes an initial value; or add (or append) these values to the key.

For larger m and/or k, independence among the hash functions can be relaxed with negligible increase in false positive rate (Dillinger & Manolios (2004a), Kirsch & Mitzenmacher (2006)). Specifically, Dillinger & Manolios (2004b) show the effectiveness of using enhanced double hashing or triple hashing, variants of double hashing, to derive the k indices using simple arithmetic on two or three indices computed with independent hash functions.

To add an element, feed it to each of the k hash functions to get k array positions. Set the bits at all these positions to 1.
To query for an element (test whether it is in the set), feed it to each of the k hash functions to get k array positions. If any of the bits at these positions are 0, the element is not in the set – if it were, then all the bits would have been set to 1 when it was inserted. If all are 1, then either the element is in the set, or the bits have been set to 1 during the insertion of other elements.

Unfortunately, removing an element from this simple Bloom filter is impossible. The element maps to k bits, and although setting any one of these k bits to zero suffices to remove it, this has the side effect of removing any other elements that map onto that bit, and we have no way of determining whether any such elements have been added. The result is a possibility of false negatives, which are not allowed.

Removal of an element from a Bloom filter can be simulated by having a second Bloom filter that contains items that have been removed. However, false positives in the second filter become false negatives in the composite filter, which are not permitted. This approach also limits the semantics of removal since adding a previously removed item is not possible.

However, it is often the case that all the keys are available but are expensive to enumerate (for example, requiring many disk reads). When the false positive rate gets too high, the filter can be regenerated; this should be a relatively rare event.


Bloom filter in:
Perl
C/C++
Ruby
Java

Oracle TOP otop

Before you can run otop.pl you will have to have an active connection to your Oracle database, meaning you will have to have Perl DBI and DBD Oracle. The installation of those 2 is described in a previous post.

[root@pubjo root]# ./otop.pl
Can't locate Curses.pm in @INC (@INC contains: /usr/lib/perl5/i386-linux /usr/lib/perl5 /usr/lib/perl5/site_perl/i386-linux /usr/lib/perl5/site_perl /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl .) at ./otop.pl line 75.
BEGIN failed--compilation aborted at ./otop.pl line 75.

To fix this problem you will need to install the Perl Module Cursus. You can download the Cursus module from CPAN.

1) Download the latest version of curses
2) Unpack curses
- [root@pubjo root]# gunzip Curses-1.15.tar.gz
- [root@pubjo root]# tar -xvf Cusres.1.15.tar
3) make/install curses
[root@pubjo root]# perl makefile.PL FORMS
[root@pubjo root]# make
[root@pubjo root]# make install
4) cleanup the tar file and the temp directory.

It could be that your installation is failing with the following error message:

c-config.h:9:21: ncurses.h: No such file or directory
Curses.c:91: parse error before `WINDOW'
Curses.c:91: warning: data definition has no type or storage class
Curses.c:94: parse error before `{'
Curses.c:96: initializer element is not constant
Curses.c:98: parse error before `return'
Curses.c: In function `c_chtype2sv':
Curses.c:142: `ERR' undeclared (first use in this function)
Curses.c:142: (Each undeclared identifier is reported only once
Curses.c:142: for each function it appears in.)
Curses.c: At top level:
Curses.c:284: parse error before `*'
Curses.c: In function `c_sv2window':
Curses.c:290: `WINDOW' undeclared (first use in this function)
Curses.c:290: `ret' undeclared (first use in this function)
Curses.c:290: parse error before `)'
Curses.c: In function `c_window2sv':
Curses.c:303: parse error before `WINDOW'
In file included from Curses.c:358:
CursesFun.c: In function `XS_Curses_longname':
CursesFun.c:3125: warning: initialization makes pointer from integer without a cast
CursesFun.c: In function `XS_Curses_touchline':
CursesFun.c:3226: `WINDOW' undeclared (first use in this function)
CursesFun.c:3226: `stdscr' undeclared (first use in this function)
CursesFun.c:3227: parse error before `int'
CursesFun.c:3233: `ret' undeclared (first use in this function)
make: *** [Curses.o] Error 1
/usr/bin/make -- NOT OK

The line “c-config.h:9:21: ncurses.h: No such file or directory” is telling you that you have to install ncurses first. The Ncurses (new curses) library is a free software emulation of curses in System V Release 4.0, and more. It uses Terminfo format, supports pads and color and multiple highlights and forms characters and function-key mapping, and has all the other SYSV-curses enhancements over BSD Curses. You can find the ncurses library project site at GNU. To find the latest version of GNU ncurses go to there ftp download site.

To install ncurses take the following steps:
1) Download the latest version of ncurses from ftp://ftp.gnu.org/pub/gnu/ncurses/
2) Unpack ncurses
- [root@pubjo root]# gunzip ncurses-5.6.tar.gz
- [root@pubjo root]# tar -xvf ncurses-5.6.tar
3) make/install ncurses
- [root@pubjo ncurses-5.6]# ./configure
- [root@pubjo ncurses-5.6]# make
- [root@pubjo ncurses-5.6]# make install

When you have successfully installed ncurses you can try to install the Perl curses again. Besides the way to install Perl curses by compiling it from the source code you can also install it by using the –MCPAN shell :

[root@pubjo root]# perl -MCPAN -e shell

This will get you into the cpan shell:

Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.7602)
ReadLine support available (try 'install Bundle::CPAN')

cpan>

Here you will also be able to install the the perl curses by executing the following command:

cpan> install Curses::Forms

If you have successfully installed al those required options you will be able to use otop and monitor you Oracle Database.

Perl and Oracle

Many people who are working with Oracle might want to have the possibility to connect to the database with Perl also many Perl coders might want to have the possibility to use an oracle database.

To be able to use Perl in combination with an Oracle database you need, besides Perl and an Oracle database, Perl DBI and DBD Oracle.

DBI is the DBI is a database access module for the Perl programming language. It defines a set of methods, variables, and conventions that provide a consistent database interface, independent of the actual database being used. It is important to remember that the DBI is just an interface. The DBI is a layer of "glue" between an application and one or more database driver modules. It is the driver modules which do most of the real work. The DBI provides a standard interface and framework for the drivers to operate within.

DBD Oracle is a Perl module which works with the DBI module to provide access to Oracle databases.

To install DBI and DBD Oracle there are 2 ways to dot this. You can use a 'perl -MCPAN -e' like way or you can download the source and build from scratch. We will use the "build from scratch" methode.

First we install DBI:

1) Make sure you are root!
2) Download DBI from CPAN
3) Use gunzip and tar to extract the archive:
gunzip DBI-1.48.tar.gz
tar -xvf DBI-1.48.tar
4) Make and install DBI:
perl Makefile.PL
make
make install
5) Cleanup all the downloaded stuff and the extracted directories.

Secondly we will build DBD Oracle:

1) Make sure you are root!
2) Download DBI Oracle from CPAN
3) Use gunzip and tar to extract the archive:
gunzip DBD-Oracle-1.16.tar.gz
tar -xvf DBD-Oracle-1.16.tar
4) Make and install DBD Oracle:
perl Makefile.PL
make
make install
5) Cleanup all the downloaded stuff and the extracted directories.

Now you should have a working Perl database interface and working oracle drivers. This means we can run a test. You could for example use this perl script

#------------------------------------------------------------------

use strict;
use DBI;
 
my $dbh = DBI->connect( 'dbi:Oracle:orcl',
                        'jeffrey',
                        'jeffspassword',
                        {
                          RaiseError => 1,
                          AutoCommit => 0
                        }
                      ) || die "Database connection not made: $DBI::errstr";

 
my $sql = qq{ SELECT id, name, title, phone FROM employees };
my $sth = $dbh->prepare( $sql );
$sth->execute();

 
my( $id, $name, $title, $phone );
$sth->bind_columns( undef, \$id, \$name, \$title, \$phone );

 
while( $sth->fetch() ) {
  print "$name, $title, $phone\n";
}

 
$sth->finish();
$dbh->disconnect();
#------------------------------------------------------------------

A script origanly done by Jeffrey William Baker, on his website there are some more details about the example.

Building Perl from scratch on Linux

Building Perl from scratch is always a good option and not really hard. You can download the source code from the Perl website. Here you will be forwarded to a FTP mirror, for example a CPAN mirror at Funet in Finland and if you like to download the latest version you have to download latest.tar.gz .

Now take the following steps:
1) Extract the archive
- [root@pubjo root]# gunzip latest.tar.gz
- [root@pubjo root]# tar –xvf latest.tar
2) Run the configuration
- [root@pubjo root]# ./Configure -de -Dprefix=/usr -Dcccdlflags='-fPIC' -Dd_dosuid -Darchname=i386-linux -Dprivlib=/usr/lib/perl5 -Darchlib=/usr/lib/perl5/i386-linux -Dsitelib=/usr/lib/perl5/site_perl -Dsitearch=/usr/lib/perl5/site_perl/i386-linux

3)Make the installation:
- [root@pubjo root]# make
- [root@pubjo root]# make test
- [root@pubjo root]# make install

This is all there is to compiling Perl from the source code. If all went without problems you will now have a working version of perl on your system. To check you can execute the following command:

- [root@pubjo root]# perl –version