Wheels Arch

Processing Multiple URLs

Up to now, we have just been processing a single URL and not really taking advantage of the batch partition.  In the last step, we update the code to take advantage of the partition.

1)      Add a getPages method to the parser interface.  This will determine how many pages a URL will span.

2)      Update the ProductParserReader to call the getPages method,  iterate over the pages, and add them to the checkpoint structure.  Note that this is a specific implementation of the page size.

3)      Using the interface, we can implement a getURLS method that takes a starting point string and add all of the URLs

4)      Now the getPartitionProperties method within ProductPartitionMapper can take advantage of this.  It will assign the URLs and add them to a partition.

5)      In the ProductPartitionMapper, change the number of partitions (this example 3).

6)      In the open method of the ProductParserReader and update it to pull out the URLs passed in for that partition

This concludes the Java Batch EE example.

Conclusion: The project is now taking advantage of the partitioning so that multiple pages can be parsed at the same time.

Next Step: Web Service Project

Writing to MongoDB

1)      Add the MongoDBJava driver as a library to your project.

2)      Create a property file to place connection information.

3)      Create properties within the property file that define the server host name, ports, and authentication information to connect to the database.

4)      Within MongoDB, switch to the products database and add a user for the parser.

5)      We will want to write the price to  the NoSQL database as a numeric as well as a string so a helper method is added to the Product class.

6)      Change the implementation in ProductParserWriter to iterate over the objects coming in and persist them to the database.  Note the key will consist of the ItemNumber, ModelNumber, and the Product number.

Conclusion: Products are now written by the writer to a MongoDB implementation.

Next Step: Processing Multiple URLs

The Parser

A parser is needed to take the HTML (doc) and turn it into the product object.  Creating an interface since we could have different parsers and not just one for the site.  This is where the jsoup library really comes into play.  The actual parser implementation is beyond the scope of the example, but will show how its created and the basic implementation (though the interface).

1)      Create the package com.crawler.batch.parsers to put the interface and its one implementation for the site.

2)      Create the interface IProdutParser

3)      The interface currently just defines one method.  getProducts sends in an HTML doc and gets back the products entity

.

4)      Create a NewEggProductParser that will implement the getProducts method from the interface.  The details of the implementation are beyond the scope of this example.

5)      Change the ProductParserProcessor to instantiate a new NewEggProductParser.  Pass in the HTML (doc) object it receives.

6)      Update the ProductParserWriter to loop through the products passed into it and log the detail information.  Also, there is a _counter variable declared to count the number of time it was called.

Conclusion: This step showed how you would use a parser within the batch framework.

Next Step: Writing to MongoDB

Create the Entities

Because we are crawling a site to get product data, it makes sense to create a product entity. Multiple products exist on one HTML page (doc) so a product container (products) will also be helpful.

1)      Create a package com.crawler.batch.entities

2)      Create a product class

3)      Fill out the Product class adding a constructor and get methods.

4)      Create a products class.

5)      The implementation of the Products class is just an array list of Product with add and get methods.  Notice that both Product and Products implement Serializable so they can be passed between batch framework components.

Conclusion: Created the product entity as well as a products container.

Next Step: The Parser

Running the Batch Job

The Java EE batch framework requires a launching mechanism either in the form of a JSP/Servlet or  an EJB timer.  We will create a JSF page to launch the job.

1)      Create a folder called resources under the web folder.

2)      And a css folder under web/resources

3)      Create a the default CSS file.

4)      Fill in the CSS file.

5)      Create a deployment descriptor called web.xml under the WEB-INF folder.

6)      Create a package called com.crawler.batch.beans and a new class file named jsfBean.java.

7)      Fill out the JSF bean so that the functions to start the batch job, get the status, and is completed method are implemented as pictured.

8)      Change the index.html default file to represent the UI to start the batch job by calling the bean startBatchJob method.

9)      Create an XHTML file called jobstarted.xhtml under the web pages folder and fill it out as pictured to show the current job status and allow functionality to update that status.

10)  Build the project (F11) and then run the project (F6)

11)  The UI will display the STARTED status once the job has been started.

12)  The check status button is available.

13)  The job will show completed when it is finished.

14)  Above shows the output of the log file showing the 6 pages that were parsed.  Nothing actually is happening here yet.  We just have a base implementation to work with.

Conclusion: The project is now able to be built and we have away to run the batch processing job.

Next Step: Create the Entities

This section will cover the partition mapper.  The partition mapper will implement the functionality to operate the batch frame work in parallel.  It is an optional component, but for a site parser it will be helpful to setup for performance.  In this section, we will just implement one URL with six pages.  We will expand this later.

1)      Create the partition mapper class (ProductPartitionMapper) in the same package as the reader, processer, and writer.

2)      Decorate the class with @Dependent and @Named.  (22,23)

3)      The ProductPartitionMapper implements PartitionMapper.   (24)

4)      Create a static logger (25)

5)      Override that mapPartitions method and return a new partition plan (28, 30)

6)      The returned partition plan overrides getPartitions and returns a 1 (37)

So far this partition mapper is setting up the partition plan with 1 partition.  This is not very useful yet, but we just want to get the core functionality working.

7)      For the partition properties, setup a new property array to the size of the number of partitions which is currently set to 1 (44)

8)      Initialize the array list and give each partition a new array list (47,50)

9)      For now, we will just hard code the first URL.  We will use internal SSDs on the newegg.com site.

10)  Put the URLs into a partition property with the tag “URLs” (60)

11)  This sets up all the URLs into partitions.  We will expand on this later. Next, lets update the reader to read in these properties (URLs).

12)  In the ProductParserReader class, add a private variable of properties.  Retrieve these parameters from the jobOberator getParameters method.

13)  Get the list of strings passed into reader via the jobOperator (partParams).  Use the “URLs” tag to retrieve the correct parameter.  (57)

14)  We also setup the page size query string that we will use later (59).  This is defaulted to 100 products per page.

15)  Within the checkpoint setup, loop through each of the URLs passed in (71).

16)  Add the URL to the check point appending the page size (74).

17)  Right now, there are 6 pages so we add a URL for each page of products (79).  This is currently hard coded to 6.  We will come back and make this dynamic later.

Conclusion: The project now has a simple partition configuration with one URL and six pages.

Next Step: Running the Batch Job

This section will cover the batch Processor and writer.  For now, this will be just a simple implementation of each so that we can build and run the project.   1)      Create a class for the processor called ProductParserProcessor.  This should go into the same package as the reader.   2)      It should implement ItemProcessor and override processItem.   3)      For now, just convert the item coming in to a Document and return the HTML of the Document object.   4)      Create the writer class called ProductParserWriter.  It should also be in the package with the reader and processor.   5)      ProductParserWriter should implement ItemWriter and needs to override open, close, writeITems, and checkpointInfo.   6)      For now, just implement writeItems to log the length of the html passed in by the processor.  Also need to return a new check point object (ProductPartsingCheckPoint) in checkpointInfo().

Conclusion: The project now has a processor that  just returns a html for the Jsoup document passed in and a writer that just logs the length of the HTML passed in.

Next Step: Partition Mapper

This section will cover the batch Reader.  The reader has two main functions of importance.  The first is the open method that will setup the checkpoint and set the iterator to the correct item.  The  second is the readItem method that will iterate through the URLs, connect to them and return the HTML doc for each item.

1)      Create a class within the com.crawler.batch.productparser oacjage baned ProductParserReader.  This is the extract portion of an ETL type job and will read each item to be processed.

2)      The product parser reader will implement the ItemReader in the batch framework.  There are several overridden methods that need to be implemented”

1. The open method is where all initialization logic will go
2. The close method is where cleanup actions like closing connections should go
3. The readItem is the main logic of this section goes and will be invoked to read each item.  It returns an object which will be passed through the batch infrastructure.

3)      Also add a reference to DI and add the Dependent and Named decorators.

4)      Additional private variables defined the checkpoint reference that we created in steps 1-6, an injected job context object, the execution ID, and Iterator object, and the logger.

5)      Within the open method, get the job operator object from the batch runtime and then pull out the execution ID from the injected job context.

6)      Change the variable named checkpoint in the constructor to chkpt for the remained of this section.  If the passed in chkpt is null then create a new checkpoint and set the item number to 0.  You can also see some INFO logging.

7)      Otherwise (if the chkpt is not null) cast the checkpoint to the checkpoint object we created in steps 1-6.

8)      Move the iterator forward until the current number is moved to the number returned by the checkpoint number.

9)      Just remove any code from the close section for now

Let’s add some plumbing for the future implementation of crawling a web site.

10)  Within the checkpoint class add an array list of strings called urls.  Within the constructor of the checkpoint, set it to an empty arraylist.

11)  Also within the check point class, implement and addURL method to add a URL to the urls list as well as the ability to get the urls.

12)  We want to setup the URLs to crawl in the open method of the reader.  We only want to do this when first creating the checkpoint.  For now, just hard coding a base URL.  We are going to use the newegg.com site in this example.  Since we will be processing the items on a page, we need to also set the page size.  While we do need to make this dynamic, for now we know that there are 6 pages for the URL that is being crawled.  Each one of these pages is added to the URL list in the checkpoint class.

13)  Now the iterator can be set to the URL iterator and moved forward to the starting point.  This really only applies in a re-start where the check point getItemNumber is not 0.

14)  Will be using a 3^rd party library called jsoup to parse HTML.  Download and add it to the project as a library.

15)  Now logic can be added to readItem.  So the iterator is set to our 6 URLs that represent the pages of the products we want to parse.  This logic will iterate through all the URLS and then connect to that URL and pull out the jsoup Document for that URL (line 120).  The checkpoint is moved forward (line 128) and the Document is returned.  This object is returned to the batch framework.

Conclusion: The project now has a reader which processes each URL and returns an HTML doc to the batch work flow.

Next Step: The processor and writer

In this section we will discuss the check point.  The check point is a serializable  object that contains that serves as the bookmark for the batch work flow.  It will be used by the reader to determine where to start the batch processing.

1)      In Netbeans chose to create a new Java package.

2)      For this project, we are going to name the package com.crawler.batch.productparser since we are going to parse products.

3)      We need a check point, so under the package created in step 2, add a class named ProductParsingCheckpoint.  The checkpoint is a serializable book mark for the batch processing to keep track of what item it has successfully processed.

4)      The check point class need to implement Serializable

5)      For the private variables, there is a version number, the itemNumber of the checkpoint and the number of items in the check point.

6)      Here is the remainder of the constructor and the methods:

1. The constructor will reset the itemNumber to 0
2. There is also a constructor that will set a specific number of items
3. A method is needed to get the current item number
4. There is a method to get and set the number of items
5. nextItem() will move forward the current item number
6. Finally, there is a method for setting the item number

Conclusion: The project now has a check point which is the plumbing for keeping track of where the batch job is processing.

Next Step: Create The Reader