endeca123.com
8Apr/1334

Installing Oracle Endeca Information Discovery 3.0

Posted by Wim Villano

So, we saw that the Quickstart Installation is no longer part of the download packages. How do we then install the whole package step-by-step?

All further documentation can be found here.

You have some different options for installations for the the presentation part (Endeca Studio). You can manually deploy it in Tomcat or you can manually deploy it in Weblogic or you can use the all-in-one package bundled with Tomcat.

Since we do not have a choice where to deploy the Endeca Server (this has to be a Weblogic application server), I also decided to deploy the Endeca Studio manually on the same Weblogic application server as the Endeca Server will be deployed. But you could save some installations steps by using the all-in-one package bundled with Tomcat.

What software do we need (Bill Of Material):

  • Oracle Endeca Server (7.5.1.0) from edelivery
  • Oracle Endeca Information Discovery Studio (3.0) for WebLogic from edelivery
  • Oracle Endeca Information Discovery Integrator (3.0) from edelivery
  • JDK version 6
    Download the Sun version here or
    On this page download the jRockit version for Windows x86-64 (recommended for Endeca Server, but not certified with Endeca Studio as Brett stated in the comments)
  • Oracle Weblogic Server 11gR1
    On this page download Weblogic 10.3.6, the Generic: 1GB release.
  • ADF Runtime
    On this page download Application Development Runtime version 11.1.1.6:

 

When you've downloaded these components you are ready to go. Perhaps first a brief summary of the steps we will do during this install, so you will not loose the overview:

  1. Install the JDK (needed for the application server)
  2. Install Weblogic Server (the application server)
  3. Install the ADF Runtime software (some additional middletier components needed for a succesful Endeca Server deployment)
  4. Install the Endeca Server software (the analytical search database)
  5. Create a Weblogic Domain for the Endeca Server
  6. Create a Weblogic Domain for Endeca Studio
  7. Update some Weblogic settings for the purpose of the installation of Endeca Studio
  8. Deploy the Endeca Studio application (the presentation layer)
  9. Install Integrator (the ETL component)
  10. Done

 

1. Install the JDK

I use the jRockit, but the Sun JDK has similar steps. To install the jRockit JDK just start the setup.exe. Click <Next> on the first screen and on the second screen enter a path. DO NOT USE SPACES!

Click <Next>. On the next page make no selections and click <Next> again. Then you are asked if you want to install the JRE as well. It is not necessary, but if you say Yes and <Next> then you are asked for a path again. Enter a different path then the previous one and click <Next>. jRockit will be installed.

2. Install Weblogic Server

To install Weblogic Server execute the download wls1036_generic.jar with the JDK. Go to the directory with wls1036_generic.jar and type:
<JDK Path>\bin\java -jar wls1036_generic.jar
Like:

The installer will start. On the Welcome screen Click <Next>.

Then select 'Create a newMiddleware Home' and enter a path where to install (I entered: d:\oracle\oeid30\fmw):

Click <Next>.

Then I try with minimum clicks :-) to make clear to the software that I do not want to be notified of updates by clicking:

Click <Next> Click <Yes> Click <Yes>

On the next screen put a check at "I wish to remain uninformed ..." and click <Continue>.

Then choose a Custom install and click <Next>. On the next screen uncheck Oracle Coherence:

and click <Next>. The just installed JDK should be checked on the screen:  . click <Next>.

Then enter a path for the installation directory:

Click <Next>. Choose not to install the Nodemanager:

Click <Next>, Click For "All users" Start Menu Folder:

Click <Next>, again <Next>. Then the install will run for about a minute. Uncheck the "Run Quickstart" and click <Done>.

Weblogic has been installed.

3. Install the ADF Runtime software

After you unzipped the downloaded software go to the directory Disk1. From a command line we start the setup.exe with a jreLoc option: setup.exe -jreLoc <JDK dir>. In my case: setup.exe -jreLoc d:\oracle\jrockit

The installer starts. On the first screen click <Next>. Select the "Skip Software Updates" option and click <Next>. After the Prerequisites Checks have been done click <Next>. Then enter the location of the Oracle Middleware Home. The just installed Weblogic home should appear in the drop down list:


Click <Next>. On the Application Server screen "Weblogic Server" is already checked. Click <Next>. After about 2 minutes the installation is finished. Click <Next> and <Finish>

4. Install the Endeca Server Software

Open a command line box, go to the unzipped Endeca Server software directory Disk1 and start the setup with the -jreLoc option as with the ADF runtime software: setup.exe -jreLoc <JDK dir>. In my case: setup.exe -jreLoc d:\oracle\jrockit

The installer starts. On the Welcome screen click <Next>. After the Prerequisites Checks have been done click <Next>.  Then select the Weblogic Middleware home from a drop down list. Leave the Oracle Home Directory to the default (EndecaServer7.5.1_1):

After entering locations click <Next>. For this installation we will not run the Endeca Server in a secure mode, so uncheck 'Yes':

Click <Next>. After 2 minutes the installation is finished. Click <Next> and then <Finish>. The core Endeca Server software has now been installed. The surrounding Web Application Server software is next.

 

5. Create a Weblogic Domain for the Endeca Server

Now we need to create a Weblogic Domain for the Endeca Server software. To do so, open a command prompt (of course you can also use Windows Explorer, but I like the command line, if something goes wrong ...) and go to the directory: <Weblogic Middleware Home>\wlserver_10.3\common\bin (for my environment: d:\oracel\oeid30\fmw\wlserver_10.3\common\bin). Type there: config.cmd

On the appearing screen choose: Create a new Weblogic domain. Click <Next>. Then select "Oracle Endeca Server - 7.5.1.0". The second check ("Oracle JRF") comes for free:

Click <Next>.

Enter a Domain name (mine: endecaserver_domain) and leave the Domain location untouched:

Click <Next>. On the next screen you enter the username and password for the administrator of the domain:

Click <Next>. Choose Production Mode and see that the right JDK already has been checked:

Click <Next>. No optional configuration. Click <Next>. Click <Create>. After 30 seconds the domain has been created. Click <Done>.

You can verify your install of the Endeca Server. Start the Endeca Server Domain by executing:
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin\startWebLogic.cmd. For my installation: d:\oracle\oeid30\fmw\user_projects\domains\endecaserver_domain\bin\startWebLogic.cmd

You will be asked to enter the username and password of the administrator of this domain (you entered this during the installation).

From a browser start this URL: http://localhost:7001/endeca-server/ws/manage?wsdl

That should give a respons something like this:

Now the Endeca Server is completely installed and operational (on port 7001). Shortly will follow a blog entry to change the port for the Endeca Server.

 

6.  Create a Weblogic Domain for Endeca Studio

For the Endeca Studio application we also need to create a domain (recommended as per the documentation). This is similar to the Endeca Server Domain creation: Open a command prompt and go to the directory: <Weblogic Middleware Home>\wlserver_10.3\common\bin (for my environment: d:\oracel\oeid30\fmw\wlserver_10.3\common\bin). Type there: config.cmd.

Select "Create a new WebLogic domain". Click <Next>. Use the default Basic domain configuration:

Click <Next>

Entera Domain name and leave the Domain location untouched:

Click <Next>, Then enter a username and password for the domain administrator:

Click <Next>. Choose Production Mode and see that the JDK has been selected.

Click <Next>.

Select modify settings of Administration Server:

Click <Next>. Since we did a basic domain installation of the Endeca Server this has been configured on port 7001. So we need to change the port of the domain where Endeca Studio will be running. Change the Listen port of the future Endeca Studio Application. I chose port 8880.

Click <Next>. Then click <Create> and after a couple of seconds the domain has been created. Click <Done>.

A domain has been created and is ready to deploy the Studio application.

7. Update some Weblogic settings for the purpose of Endeca Studio

Before you can deploy the Endeca Studio application, we need to modify some Weblogic settings. Edit the file:
<Weblogic Middleware Home>\user_projects\domains\<Endeca Studio Domain>\bin\setDomainEnv.cmd

Copy the italics text below and paste it in the top of the file, i.e. just after set WL_HOME.

set JAVA_OPTIONS=-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl

In the same file update (2 times) memory arguments:
Change set WLS_MEM_ARGS_64BIT=-Xms256m -Xmx512m into: set WLS_MEM_ARGS_64BIT=-Xms256m –Xmx1024m
and Change
set MEM_MAX_PERM_SIZE_64BIT=-XX:MaxPermSize=256m into: set MEM_MAX_PERM_SIZE_64BIT=-XX:MaxPermSize=512m

Save the file.

Stop the Endeca Studio domain if started (via command <Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin\stopWebLogic.cmd)  and start it (via command <Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin\startWebLogic.cmd).

8. Deploy the Endeca Studio application

Now you will deploy the Endeca Studio application in the just created Weblogic domain. After you unzip the edelivery Studio package for Weblogic you'll see some files:

Copy the files portal-ext.properties and endeca-portal-weblogic-3.0.10089.ear to <Weblogic Middleware Home>\user_projects\domains

In the directory <Weblogic Middleware Home>\user_projects\domains create the following directories: data, deploy and weblogic-deploy. In the newly created directory <Weblogic Middleware Home>\user_projects\domains\data create the directory endeca-data-sources.

The directory <Weblogic Middleware Home>\user_projects\domains will look like:

Now let's deploy the Endeca Studio file (endeca-portal-weblogic-3.0.10089.ear).

Make sure your Endeca Studio domain has been started. Open a browser and go to URL: http://localhost:<port number>/console (for me: http://localhost:8880/console).

Go to Configure applications:

Click Lock & Edit:


Click Install:

Locate the file endeca-portal-weblogic-3.0.10089.ear in <Weblogic Middleware Home>\user_projects\domains and paste the path to that file in the Path space. Hit <enter>:

The file endeca-portal-weblogic-3.0.10089.ear appears. Select it and click <Next>.
Leave the defaults:

Click <Next>.
Again leave the defaults:

Click <Finish>. After the deployment is finished you will return to the overview screen. Click on <Activate Changes>:

Then the message appears that no restart is necessary:

Then click on deployments in the left pane of the Domain Structure. In the deployments then check endeca-portal-weblogic and from the Start drop down select: Servicing All Requests:

On the next screen click <Yes>. When the deployment screen returns and the State has been changed to Active, you can check the URL: http://localhost:<port number> (in my case: http://localhost:8880):

You can log on with username: admin@oracle.com and password: Welcome123. You have to reset the password immediately after log on.

That was the install of Endeca Studio.

9. Install Integrator

The last piece of software before we can create our projects is the Integrator. After the unzip of the downloaded file you will see 3 files. 2 Files are needed when you would want to deploy the server version for production environments. For now double click the file EID_3.0_Integrator.exe.

On the opening screen click <Next>, click <Next> again. Then enter a location:

Click <Next>, Click <Next> and click <Finish>.

That was the Integrator install.

10. Done

Hopefully the next release of Endeca will give us the 'next, next, finish' experience again ...

Enjoy Endeca 3.0

 

 

11Sep/124

Clean Up an Endeca Data Store

Posted by Wim Villano

This article was written at the time of EID 2.3. Since then some changes have been made to EID.
Some additional remarks for Endeca v3.0:
- the endeca-cmd commands now have a suffix of -dd (data domain) in stead of -ds (data store). So endeca-cmd list-ds is now in v3.0: endeca-cmd list-dd

 

With all tests and tryouts I keep creating new data stores. Every now and then I have to look up again how to clean up those data stores. So just for my record to find this quick I'll post the commands here.

All commands are done via the batch file endeca-cmd.bat that you will find in the directory: <Install Dir>\Endeca\Server\2.3.0\endeca-cmd\

To get a list of current active data stores, type:
endeca-cmd list-ds

Endeca List Active Data Stores

To stop a Data Store, type:
endeca-cmd stop-ds FirstSteps

To detach a Data Store, type:
endeca-cmd detach-ds FirstSteps

And then you can safely delete the Index directory files. Default location of the index files: <Install Dir>\Endeca\Server\2.3.0\endeca-server\data

2Sep/127

Enable Snippeting

Posted by Wim Villano

Hi there.

After a holiday break I want to finalize some blog posts. Enable Snippeting was not the first I wanted to finish, but since there was a question about how to enable this, I'll start with this one.

From the documentation about snippeting: "The snippeting feature provides the ability to return an excerpt from a record — called a snippet — to an application user who performs a record search query." .  So snippeting makes it easier for users to read search results at a glance and could help users to go faster from search to information.

If you for example would search for 'mountain'  in the QuickStart application ...
The default result of the search for the attribute 'Survey Response' will be like this for the Results Table and Results List:
OEID Results with no snippeting

The complete text of the surveys.

With snippeting enabled it will look like this:
OEID Results with snippeting

As you can see the search term is highlighted and it is surrounded by a number of words (in this case 10 in total).

How can we enable this?
In the documentation we see the code to enable snippeting:

OEID code to enable snippeting

There are 3 parameters to fill in: The Search Interface name (here: MySearch), Snippet Size (number of surrounding words, here: 10) and the attribute name (here: Description). We will change these to fit the requirement for the attribute SurveyResponse in the Quickstart application.

This code needs to be sent to the Oracle Endeca Server using the Configuration Web Service or Integrator. We will use the Integrator.

Probably we have mentioned it before on the blog, but the QuickStart application can be a great source for re-usable components in your own projects. The same goes for enabling snippeting.

We take the graph LoadIndexingConfiguration.grf from the Quickstart application as the base:

OEID Integrator base graph Create Search Configuration

This is the graph in which you should add the correct code in the component: Create Search Interface (surrounded by a green rectangle). For now we take a short cut and copy the component: Load Search Interfaces (surrounded by a red rectangle) to a new empty graph e.g. Snippet.grf:

OEID Integrator Web Service Component

If you now double click on the component Load Search Interfaces and then open the Request Structure, you will see that the RECSEARCH_CONFIG is used here:

OEID Integrator Webservice Configure to enable snippeting

Now we replace the $xmlString with our wanted code:
<SEARCH_INTERFACE NAME="All">
<MEMBER_NAME SNIPPET_SIZE="10">SurveyResponse</MEMBER_NAME>
</SEARCH_INTERFACE>

As search_interface name we enter "All" (there are more search interfaces defined in the Quickstart application, also: Employees, Resellers and Survey. For now we only go with "All"). The snippet_size is set to 10. Which means that the snippet which will be returned contains 10 words max. And the attribute we want to snippet is SurveyResponse.

Save it and run the graph. You will see immediately the result in the Studio Application when you type a search term on the page Products. Because we now did not take all Search Interfaces and Parameters into account as defined in the graph LoadIndexingConfiguration.grf the search interfaces are a bit screwed (e.g. when you create a new search box only the 'All' search interface will be available instead of the 4 mentioned above). To correct it just run the graph LoadIndexingConfiguration.grf again.

These are the basics to enable snippeting.

Check the  documentation to find out more about the behaviour e.g.: Snippeting works for the Results List, Results Table and Data Explorer and can be set for an attribute per Search Interface, etc.

 

Just for completeness. If you want to do it more the right way, change the component Change Search Interface in the graph LoadIndexingConfiguration.grf. You could change the code as shown below or also (even better) solve it like mentioned in the comment by Ricardo:

Standard Code no snippeting

to something like:

Enable Snippeting Code

 

 

7Aug/1210

Add a geospatial attribute to Endeca Server to use the Map component

Posted by Elio Idema

Maps are an attractive way of visualizing your data. The 'Map' component of EID is surprisingly easy to configure. The prerequisite is that the data in your datastore contains a longitude/latitude attribute. This is also not very difficult to achieve. First, let's add a map to the EID portal and see what happens...

Studio Configure Map Preferences

Select 'Preferences' to observe that your Endeca Server data source must contain a valid geospatial attribute in order to use the map.

Studio Configure Map Preferences

The geospatial attribute is a data type in Endeca Server that is designated to storing longitude/latitude combinations. See the Data Loading Guide for the list of data types (property types) that are supported. In this case we are interested in the 'mdex:geocode' type.

Now, let's go to Integrator where I for example have data that contains attributes called 'X' and 'Y', representing the latitude and longitude as a number. Here's a snap of the metadata.

Metadata Geo Codes

Before we can feed this data to Endeca Server as a geospatial attribute, we have to make sure that it is transformed to a string with format '<latitude> <longitude>' (the two decimal numbers with a space in between). In order to achieve this we insert a transformation to our graph. The whole graph looks like this.

OEID Integrator sample graph update/inserting records

Suppose we call the new attribute 'LATLONG', the code for the transformation could look like this.

The expressions map the input fields X and Y onto the output field LATLONG. The example contains some checks for null values, but the actual conversion is done in the following expression.

$out.0.LATLONG = num2str($in.0.X,"#.#####") + " " + num2str($in.0.Y,"#.#####");

The 'num2str' function converts the decimal numbers to a 5-digit string. The '+' sign is the concatenation operator. Finally, note the space string that is put between the numbers.

So, how does Endeca Server know that the LATLONG attribute is of type 'mdex:geocode'? It doesn't. By default, it will be interpreted as text.

In order to set the property type, edit the metadata of the second edge, navigate and click on the LATLONG field. (If the field does not yet exist, add it.) Add a custom property by clicking on the green + sign on the right-hand side of the dialog.

In the next dialog enter the following values.

OEID Integrator Add metadata field

Confirm all your changes, run your graph. Go back to the front-end and observe the results in the map component. After pressing the 'Update data source' button, your map component will no longer show the error message and you are able to select the LATLONG attribute as the 'Geospatial filtering attribute'.

Happy painting!

 

12Jul/1210

Enable spelling correction and Did You Mean?

Posted by Elio Idema

This way of enabling Spelling Correction works up to v2.4

 

A mindblowing feature of Endeca (at least the first 50 times you see it) is the spelling correction and the so-called 'Did You Mean?' functionality. It comes out of the box and operates based on the data you've loaded into Endeca Server. Yes, this means no messing around with dictionary files!

So what is it?

  • Spelling Correction - corrects typos when you use the search box. This results in an auto-correction of your search term. If your search term does not give any results and there is a term available in your data that looks like your term, the look-a-like will be used.
  • Did you mean? - suggests terms that will have more resulting records. A click on the suggested term will change your search.

A prerequisite for these features is that you have defined a search interface, but then enabling them is very simple. After you've loaded your data, just navigate to the following url:

http://localhost:7770/admin/<your datastore name here>?op=updateaspell

Check the results, it's fun! Here are some examples (hint: one of these results is fake ;))

The spelling dictionary must be updated after each full load...You don't want to navigate to the url manually every time after an etl run, so let's embed it in a graph.

Add a 'HTTP connector' component from the palette to your graph (don't forget to set the phase correctly if needed to make sure the call is done after data loading into Endeca Server).

Edit its properties and enter the url in the URL property.

Variables are allowed as well, so something copy in something like this:

http://${ENDECA_SERVER_HOST}:${ENDECA_SERVER_PORT}/admin/${DATA_STORE_NAME}?op=updateaspell

Wow, your EID app has just become even more amaaaaazing!

22Jun/120

Adding a multi value attribute to Endeca Server

Posted by Elio Idema

Some of the elements in Integrator will output multiple values in a single field. These values are separated by a special character. Examples are the 'Text Tagger' and the 'Text Enrichment' components. But it may also be present in your source data. Endeca Server is able to handle this and split the field into multiple fields.

First, an example. The following image shows a graph with a database table that is directly inserted into Endeca Server. (If you don't know how to set this all up, look here.) The data is also shown, notice that the field 'SKILLS' contains multiple values.

Now, let's observe the result in Studio after running the graph like this. As expected, the column SKILLS is loaded exactly as it is supplied.

However, the nice thing about Endeca is that it is able to store the same attribute multiple times on the same record. The solution is easy, just tell Integrator what the delimiter is by setting it as a property of the 'Add/Update Records' object.

After running the graph now, the results look much nicer. We can now see that there are 4 people with the skill 'OBI', 3 with the skill 'EID', and so on. This is very useful for filtering.

The 'Record Details' component shows the results somewhat better.

In a lot of cases it is probably useful to add some extra metadata to this column. For instance, setting its property 'system-navigation_Select' to 'multi-or' can be useful. More on setting attribute metadata is posted here.

29Mar/120

Installing Endeca Server as a Windows service

Posted by Elio Idema

This article was written at the time of EID 2.2.2. Since then changes have been made to EID which may have made some information in this article outdated for the most recent version of EID. Use this information with proper judgement.

The first question any Windows administrator will ask when they are asked to install a server product is: Will it run as a service? For Endeca Server (formerly known as MDEX) the answer is yes. And it is clearly documented in chapter 8 of the Administrator's Guide (AdminGuide.pdf). But... for those who are lazy, you can also take the following steps ;)

  • Download endeca_server_service_scripts.zip and save it on the Endeca Windows machine.
  • Unzip it.
  • Adapt the variables in 'endeca_server_service_create.bat' to your needs.
  • Make sure there is no Endeca Server running on the machine.
  • Run the script!
  • Start the service 'Oracle Endeca Server' from the Windows Services screen.
  • You are done :)

The contents of the scripts are posted below. Have fun!

endeca_server_service_create.bat

rem Script that creates a Windows service for the Endeca Server
rem Visit http://www.endeca123.com/installing-endeca-server-as-a-windows-service for more info
rem www.endeca123.com

rem VARIABLES, adjust to personal flavour
set install_dir=c:\Endeca\Latitude\2.2.2\MDEX
set index_dir=c:\mdex_db
set mdex_port=5555
set mdex_bulk_port=5556
set mdex_extra_args=

rem REMOVE OLD DB FILES
rd /q /s "%index_dir%\mdexdb_indexes"
del /q "%index_dir%\mdexdb.worddat"
del /q "%index_dir%\dgraph.log"
del /q "%index_dir%\dgraph.pid"
del /q "%index_dir%\dgraph.reqlog"

rem CREATE NEW DB
call "%install_dir%\bin\mkmdex" --port "%mdex_port%" "%index_dir%\mdexdb"

rem CREATE SERVICE
sc create EndecaServerService displayname= "Oracle Endeca Server" type= own error= severe binpath= "%install_dir%\bin\dgraph.exe --port %mdex_port% --bulk_load_port %mdex_bulk_port% --threads 4 --pidfile %index_dir%\dgraph.pid --log %index_dir%\dgraph.log --out %index_dir%\dgraph.out %mdex_extra_args% %index_dir%\mdexdb"

endeca_server_service_delete.bat

rem @echo off
rem Script that deletes a Windows service for the Endeca Server
rem Visit http://www.endeca123.com/installing-endeca-server-as-a-windows-service for more info
rem www.endeca123.com

rem DELETE SERVICE
sc delete EndecaServerService

29Mar/124

Endeca Server web services

Posted by Elio Idema

This article was written at the time of EID 2.2.2. Since then changes have been made to EID which may have made some information in this article outdated for the most recent version of EID. Use this information with proper judgement.

Once you've got the Endeca Server up and running (either from Clover or from the command line) you might wonder how to interact with it. The answer is simple: through SOAP web services. Let's dig into this a little bit...

Point your browser to the following url to get a list of available webservices.

http://localhost:5555/ws/

Now, each webservice in the list that is returned can be used through the WSDL description by using the URLs in the first column of the table below. The second column tells you where to find each service in the product documentation. The third column points to the API documentation in your installation directory (assuming you've installed the quickstart package).

Link Documentation API
http://localhost:5555/ws/admin?wsdl AdminGuide.pdf, Chapter 2 <install dir>\MDEX\doc\wsdl\admin\index.htm
http://localhost:5555/ws/config?wsdl DevGuide.pdf, Chapter 3 <install dir>\MDEX\doc\wsdl\config\index.htm
http://localhost:5555/ws/config_read_only?wsdl DevGuide.pdf, Chapter 3 <install dir>\MDEX\doc\wsdl\config\index.htm
http://localhost:5555/ws/conversation?wsdl DevGuide.pdf, Chapter 4 <install dir>\MDEX\doc\wsdl\conversation\index.htm
http://localhost:5555/ws/ingest?wsdl DataIngestGuide.pdf <install dir>\MDEX\doc\wsdl\ingest\index.htm
http://localhost:5555/ws/sconfig?wsdl
http://localhost:5555/ws/transaction?wsdl DevGuide.pdf, Chapter 5 <install dir>\MDEX\doc\wsdl\transaction\index.htm

Note: I did not find a good reference to what the 'sconfig' service really does. Inspecting the WSDL gives a hint: 'operations to be performed for configuring Semantic Entities'. Stuff for later.

Now, time for some action, let's load some data through the ingest service...

The webservice interface is generic and the tools in CloverETL are basically wrappers around these services. This means that not only Clover, but any tool that is able to 'speak' SOAP can be used to manipulate and use data in the Endeca Server. So, I would like to try working with the SoapUI tool. SoapUI is a free tool for working with and testing SOAP-based web services.

The first thing to do after starting SoapUI is to create a new project based on the ingest WSDL.

Give your project a name and supply the WSDL url.

SoapUI has now created a project based on the WSDL. Expand the 'ingestRecords' operation and double-click on the generated 'Request1' to open and edit the SOAP message. See how SoapUI has extracted from the WSDL what the message for this operation should contain, which are variables, and which parts are optional. Variables are indicated by a question mark '?'. Note that the 'ingestRecords' operation allows you to add records but also to delete records from the Endeca Server.

Edit the message to remove some of the optional parts. Add a primary key and add some records to be added. Or simply copy the message below ;)

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.endeca.com/MDEX/ingest/2010" xmlns:ns1="http://www.endeca.com/MDEX/XQuery/2009/09">
<soapenv:Header/>
<soapenv:Body>
<ns:ingestRecords>
<ns:primaryKeys>
<ns:primaryKey name="EMPNO"/>
</ns:primaryKeys>
<ns:addAssignments>
<ns1:record>
<EMPNO type="mdex:int">7369</EMPNO>
<ENAME type="mdex:string">SMITH</ENAME>
<JOB type="mdex:string">CLERK</JOB>
<SAL type="mdex:int">800</SAL>
</ns1:record>
<ns1:record>
<EMPNO type="mdex:int">7499</EMPNO>
<ENAME type="mdex:string">ALLEN</ENAME>
<JOB type="mdex:string">SALESMAN</JOB>
<SAL type="mdex:int">1600</SAL>
</ns1:record>
<ns1:record>
<EMPNO type="mdex:int">7521</EMPNO>
<ENAME type="mdex:string">WARD</ENAME>
<JOB type="mdex:string">SALESMAN</JOB>
<SAL type="mdex:int">1250</SAL>
</ns1:record>
</ns:addAssignments>
</ns:ingestRecords>
</soapenv:Body>
</soapenv:Envelope>

Press the 'Submit' button on the top-left of the 'Request1' window in order to submit your SOAP request. Observe the resulting message back from the Endeca Server in the right-hand side pane. The message shows that 4 properties have been created, these are the 4 fields EMPNO, ENAME, JOB and SAL. The message also shows that 3 records have been inserted.

By taking these steps, you've now manually inserted some sample EMP data into the engine. Of course it's much easier to do this using the Latitude Data Integrator, as described in the First Steps blogpost. The First Steps post also describes how to get to this data from the Latitude Studio front-end. Here's what the result looks like using the records you've just inserted.

9Mar/120

Running the Endeca Server from the command line

Posted by Elio Idema

This article was written at the time of EID 2.2.2. Since then changes have been made to EID which may have made some information in this article outdated for the most recent version of EID. Use this information with proper judgement.

After all the fun I had running the server using the examples from the quickstart Clover project, I got curious about this executable called 'dgraph.exe'. This is the actual executable of the Endeca Server, the search/database engine that is at the core of EID. How does it work? How can I run it? Where does it store its data? Let's see...

Running 'dgraph.exe' from the command line gives the usage.

All options between [ square brackets ] are optional, so it looks like only the '<db_prefix>' is mandatory. This is the place where the index is stored on disk. Entering a non-existing directory will give an error.

As the error message tells us, the index can be created with 'mkmdex.cmd' which is also found in the bin directory. Running this script also gives its usage details.

The file 'mkmdex.cmd' is a script that calls 'mkmdex-helper.js'. For some reason, this script appends '_indexes' to the directory that was supplied and creates this directory. The script starts 'dgraph.exe', calls one of the dgraph webservices to ingest some default settings in the newly created index and then stops dgraph again. Nothing to really care about, but funny to notice. The end-result is important; a new index is created!

Now it is time to run dgraph on our fresh index. Notice that the directory I supply as the argument does not actually exist! The dgraph process also appends '_indexes'. It seems a bit odd to me, but it works :)

Endeca Server is now running and it has started a HTTP server on a default port 5555. Let's have a look over there. Entering localhost:5555 in your browser will give a 404 error, go to the /admin extension instead.


The screen shows some operations. They can be called by navigating to '/admin?op=' followed by the admin operation you would like Endeca Server to execute. The following are funny to try:

http://localhost:5555/admin?op=ping
http://localhost:5555/admin?op=stats
http://localhost:5555/admin?op=exit

The last one really stops the 'dgraph.exe' process, check it out in your command prompt... Interesting from a security point-of-view, huh??? :)

So, that's where this dgraph fun ends. Of course, in real life, it is probably better to supply proper --out and --log arguments to the dgraph process and to run the process in the background. The quickstart example startup scripts already do this, so have a look at them... I guess it is also a good idea to install the Endeca Server as a service. Food for thought... See you next time!

8Mar/120

Running the Endeca Server from Clover

Posted by Elio Idema

This article was written at the time of EID 2.2.2. Since then changes have been made to EID which may have made some information in this article outdated for the most recent version of EID. Use this information with proper judgement.

If  you’ve installed the quickstart you have the sample CloverETL project ‘Latitude_QuickStart’ installed. This project contains a lot of useful sample graphs that help in doing the basic plumbing of the Endeca Server (or its old name 'MDEX engine', or its older name 'Dynamic graph server'). Let’s have a look a them and get a server up and running.

First of all, at some point with every software product, you’ll want to use variables. The quickstart project provides the ‘workspace.prm’ file in the root of your project. It contains a set of useful variables that can be used pretty much anywhere in your ETL definitions. Look at them, there are some that are specifically interesting right now.

The MDEX_* parameters are obviously the location of the Endeca Server (MDEX engine). This enables the ETL components to connect and write to the server. Within the quickstart project these are also the parameters used to start the Endeca server. Let’s see how this is done.

Assuming you are on Windows, open the ‘InitMDEXWindows.grf’ graph and you’ll find a single component called ‘Init MDEX’ which is of the type ‘SystemExecute’. A double-click on it will show that this calls a script called ‘create_dgraph_if_none.bat’, which is located in the ‘script’ directory of the workspace.

This is the full command:

${SCRIPT_DIR}/create_dgraph_if_none.bat ${MDEX_INSTALL_DIRECTORY} ${MDEX_INDEX_DIRECTORY} ${MDEX_HOST} ${MDEX_PORT} ${MDEX_BULK_PORT} ${MDEX_EXTRA_ARGS}

The call uses the variables and it contains the arguments that are needed to start the MDEX engine. You can run the graph and it will fire an Endeca Server engine. Try it, check that your graph finishes successfully and check if the ‘dgraph.exe’ process is running from your taskmanager.

Inspecting the 'create_dgraph_if_none.bat' file reveals that it checks if a 'dgrap.exe' process is running. If it is, the script quits. If it is not, the script will delete any existing data in the ${MDEX_INDEX_DIRECTORY}, create an empty index at that location and then start the engine. This makes this graph especially useful during development of ETL. It makes sure a fresh Endeca Server is running and it is harmless to accidently run it more than once.

Another useful graph during development is 'ResetMDEX.grf'. I guess it's self-explanatory. Happy data loading!