If you saw my previous post you could have seen that I use some of the components from the GettingStarted project. Now the 3.1 has been released on OTN (so you do not have to change those 3.0 components, but just copy the new ones into your project):
So we see a lot of new exciting features in v3.1 on slides, blogs and youtube, but what about my current (demo) projects? Since it is not possible to import indexes of v3.0 into v3.1 you have to run the graphs again. So that is what I tried. Below you will find my experiences for a small demo project and the changes you have to do before it will run.
In most of my projects I re-use the InitDataDomain graph and the LoadConfiguration graph of the GettingStarted project of OEID v3.0. Both did not work anymore in v3.1.The other component which now needs additional configuration is the Text Tagger RegEx. And there is a new concept introduced: Collection Key with the data ingest components.
Changes in the graph InitDataDomain
Oracle uses in v3.1 a higher version of some webservice calls, so to get this graph up and running again you have to replace the following strings in the components (Get Data Domain Status, Enable Data Domain and Create Data Domain) used in the graph:
http://www.endeca.com/endeca-server/manage/1 with: http://www.endeca.com/endeca-server/manage/2
http://www.endeca.com/endeca-server/manage/1/0 with: http://www.endeca.com/endeca-server/manage/2/0
Changes in the graph LoadConfiguration
In the LoadConfiguration graph replace the following strings in the components:
http://www.endeca.com/MDEX/config/services/config/2 with: http://www.endeca.com/MDEX/config/services/config/3
http://www.endeca.com/MDEX/config/services/types/2/0 with: http://www.endeca.com/MDEX/config/services/types/3/0
Data Ingest Components (Bulk Load and Merge Records)
Furthermore a new concept has been introduced: Collection Key. Seems that some management/maintenance can be done on a collection. You can just give it a new name. You will see this name as the Data Set Name in Studio.
Text Tagger Regex
The Text Tagger Regex now requires to have the regular expression added to the component via the input port.
Perhaps nothing shocking here, but be aware that the column which contains the regular expression is called SearchPattern in the metadata edge, that was not immediately clear to me from the documentation:
Also excited about the new features like self service enrichment in the Studio component? But you do not see it as in the Youtube Videos?
Then you probably are using the wrong JDK version. It seems only to be working with JDK6 and not JDK7. Of course JDK6 is a pre-requisted according to the documentation ....
Installation of the software with JDK6 results in the right tabs:
Hi All, Endeca Information Discovery 3.1 is available for downloading from edelivery.oracle.com.
It promises to be release with a lot of new features with a strong focus on self-service. But also if we look at the installable files we can see an new Web Acquisition Tool and Install scripts.
In the meantime you can watch some new videos on the Endeca Youtube channel for some first impressions. Search for "OEID 3.1". Especially the "Warranty Self-Service Demo" shows some new capabilities and visualizations.
If time permits I will get back soon to you to share some experiences.
One completely new feature of OEID 3.0 is the Provisioning Service. What is that? The Provisioning Service provides the business user a way to do self-service data loads and application creation. It provides a framework where users can upload spreadsheet data (from a desktop) and create a discovery application on top of that. If we read the documentation carefully we see a Note: In Version 3.0, the Provisioning Service only supports upload of Excel files. This indicates, I think, that more sources will be added in the future ...
Where is the button to upload a spreadsheet? Well, first we need to install the Provsioning Service. It runs as a web application in a WebLogic Server container. There is an expert install (see the documentation) and an easy (test/development) install. Below you will find the steps involved with an easy install.
I am assuming you already have done a basic install of OEID 3.0 as described in this post (or similar) and downloaded and unzipped the Provisioning software. First you need to create an additional domain via the provided eidProvisioningTemplate.jar. Start config.cmd located in <middleware home>\wls_server_10.3\common\bin.
Create a New WebLogic Domain
Select the template eidProvisioningTemplate.jar from the location where you unzipped the package.
Leave all the settings to defaults
Enter the Administrator credentials
Select the Production Mode
Check Administration Server (to be able to change the Listen port)
Change the Listen port and uncheck SSL enabled
The next thing is to change the plan.xml. You will find this in <middleware home>\user_projects\domains\<provisioning domain>\eidProvisioningConfig.
Change the following 4 parameters:
The last change is for the upload directory. During the upload of the spreadsheet file it gets temporary stored by the Provisioning Service. By default the service uses the operating system temporary directory. You can also specify a directory. That directory is relative to the domain home directory. I want to specify my own directory and create a directory in my Provisioning Service directory structure. My domain is: D:\oracle\oeid30\fmw\user_projects\domains\oracle.eid-ps and I create the upload directory there:
Save the file plan.xml. Start the Provisioning Domain (<Middleware Home>\user_projects\domains\<provisioning domain>\bin>startWebLogic.cmd)
Up to now you are not using SSL, so only the port and server are required of the Provisioning Service:
Click <Save>. Go back to the Home page.
Be aware: If the Provisioning Service domain has not been started, this option will not show up!
Self-Service for everybody!
As with release v2.3 Oracle has put some screencasts on Youtube: OEID v3.0 Screencasts. Very nice screencasts to get familiar with v3.0 and do basic development & configuration.
In the blog entry Installing Oracle Endeca Information Discovery 3.0 the Endeca Server was installed on the default port of 7001. If you have running multiple installations on your machine (i.e. Oracle BI) which also makes use of this port you probably want to change it.
Here are the steps to do so:
Start the Endeca Server (
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin\startWebLogic.cmd) and log in on the console. Click on Environment -> Servers and then in the right pane Adminserver:
Click on the <Lock & Edit> button and change the Listen Port to i.e. 7770 (can be any free port number):
Click on <Save>
Click <Activate Changes>
Stop the Endeca Server (
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin\stopWebLogic.cmd).
Now the port of the Webservice interface has been changed, but (unfortunately) there are also some hard coded port numbers in some files.
These files need to be changed:
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\config\EndecaCmd.properties
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\config\EndecaServer.properties
The port is also hard coded in the stopWeblogic.cmd found in directory
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin
set ADMIN_URL=t3://<your host name>:7001 to
set ADMIN_URL=t3://<your host name>:7770
With the above changes you can use the endeca-cmd.bat located in directory <Weblogic Middleware Home>user_projects\domains\<endeca server domain name>\EndecaServer\bin.
But the endeca-cmd.bat located in <Weblogic Middleware Home>\EndecaServer7.5.1_1\endeca-cmd (which is actually called by the one mentioned in the previous sentence :-) ) is missing the environment variable ENDECA_CMD_PROPERTIES (I could not find where that default port is stored ....).
So if you also want to use this endeca-cmd.bat I suggest to add the environment variable ENDECA_CMD_PROPERTIES to the .bat file. You can copy it from the file <Weblogic Middleware Home>user_projects\domains\<endeca server domain name>\EndecaServer\bin\endeca-cmd.bat.
So, we saw that the Quickstart Installation is no longer part of the download packages. How do we then install the whole package step-by-step?
All further documentation can be found here.
You have some different options for installations for the the presentation part (Endeca Studio). You can manually deploy it in Tomcat or you can manually deploy it in Weblogic or you can use the all-in-one package bundled with Tomcat.
Since we do not have a choice where to deploy the Endeca Server (this has to be a Weblogic application server), I also decided to deploy the Endeca Studio manually on the same Weblogic application server as the Endeca Server will be deployed. But you could save some installations steps by using the all-in-one package bundled with Tomcat.
What software do we need (Bill Of Material):
- Oracle Endeca Server (18.104.22.168) from edelivery
- Oracle Endeca Information Discovery Studio (3.0) for WebLogic from edelivery
- Oracle Endeca Information Discovery Integrator (3.0) from edelivery
- JDK version 6
Download the Sun version here or
On this page download the jRockit version for Windows x86-64 (recommended for Endeca Server, but not certified with Endeca Studio as Brett stated in the comments)
- Oracle Weblogic Server 11gR1
On this page download Weblogic 10.3.6, the Generic: 1GB release.
- ADF Runtime
On this page download Application Development Runtime version 22.214.171.124:
When you've downloaded these components you are ready to go. Perhaps first a brief summary of the steps we will do during this install, so you will not loose the overview:
- Install the JDK (needed for the application server)
- Install Weblogic Server (the application server)
- Install the ADF Runtime software (some additional middletier components needed for a succesful Endeca Server deployment)
- Install the Endeca Server software (the analytical search database)
- Create a Weblogic Domain for the Endeca Server
- Create a Weblogic Domain for Endeca Studio
- Update some Weblogic settings for the purpose of the installation of Endeca Studio
- Deploy the Endeca Studio application (the presentation layer)
- Install Integrator (the ETL component)
1. Install the JDK
I use the jRockit, but the Sun JDK has similar steps. To install the jRockit JDK just start the setup.exe. Click <Next> on the first screen and on the second screen enter a path. DO NOT USE SPACES!
Click <Next>. On the next page make no selections and click <Next> again. Then you are asked if you want to install the JRE as well. It is not necessary, but if you say Yes and <Next> then you are asked for a path again. Enter a different path then the previous one and click <Next>. jRockit will be installed.
2. Install Weblogic Server
To install Weblogic Server execute the download wls1036_generic.jar with the JDK. Go to the directory with wls1036_generic.jar and type:
<JDK Path>\bin\java -jar wls1036_generic.jar
The installer will start. On the Welcome screen Click <Next>.
Then select 'Create a newMiddleware Home' and enter a path where to install (I entered: d:\oracle\oeid30\fmw):
Then I try with minimum clicks :-) to make clear to the software that I do not want to be notified of updates by clicking:
|Click <Next>||Click <Yes>||Click <Yes>|
On the next screen put a check at "I wish to remain uninformed ..." and click <Continue>.
Then enter a path for the installation directory:
Click <Next>. Choose not to install the Nodemanager:
Click <Next>, Click For "All users" Start Menu Folder:
Click <Next>, again <Next>. Then the install will run for about a minute. Uncheck the "Run Quickstart" and click <Done>.
Weblogic has been installed.
3. Install the ADF Runtime software
After you unzipped the downloaded software go to the directory Disk1. From a command line we start the setup.exe with a jreLoc option:
setup.exe -jreLoc <JDK dir>. In my case:
setup.exe -jreLoc d:\oracle\jrockit
The installer starts. On the first screen click <Next>. Select the "Skip Software Updates" option and click <Next>. After the Prerequisites Checks have been done click <Next>. Then enter the location of the Oracle Middleware Home. The just installed Weblogic home should appear in the drop down list:
Click <Next>. On the Application Server screen "Weblogic Server" is already checked. Click <Next>. After about 2 minutes the installation is finished. Click <Next> and <Finish>
4. Install the Endeca Server Software
Open a command line box, go to the unzipped Endeca Server software directory Disk1 and start the setup with the -jreLoc option as with the ADF runtime software:
setup.exe -jreLoc <JDK dir>. In my case:
setup.exe -jreLoc d:\oracle\jrockit
The installer starts. On the Welcome screen click <Next>. After the Prerequisites Checks have been done click <Next>. Then select the Weblogic Middleware home from a drop down list. Leave the Oracle Home Directory to the default (EndecaServer7.5.1_1):
After entering locations click <Next>. For this installation we will not run the Endeca Server in a secure mode, so uncheck 'Yes':
Click <Next>. After 2 minutes the installation is finished. Click <Next> and then <Finish>. The core Endeca Server software has now been installed. The surrounding Web Application Server software is next.
5. Create a Weblogic Domain for the Endeca Server
Now we need to create a Weblogic Domain for the Endeca Server software. To do so, open a command prompt (of course you can also use Windows Explorer, but I like the command line, if something goes wrong ...) and go to the directory:
<Weblogic Middleware Home>\wlserver_10.3\common\bin (for my environment:
d:\oracel\oeid30\fmw\wlserver_10.3\common\bin). Type there:
On the appearing screen choose: Create a new Weblogic domain. Click <Next>. Then select "Oracle Endeca Server - 126.96.36.199". The second check ("Oracle JRF") comes for free:
Enter a Domain name (mine: endecaserver_domain) and leave the Domain location untouched:
Click <Next>. Choose Production Mode and see that the right JDK already has been checked:
Click <Next>. No optional configuration. Click <Next>. Click <Create>. After 30 seconds the domain has been created. Click <Done>.
You can verify your install of the Endeca Server. Start the Endeca Server Domain by executing:
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin\startWebLogic.cmd. For my installation:
You will be asked to enter the username and password of the administrator of this domain (you entered this during the installation).
From a browser start this URL:
Now the Endeca Server is completely installed and operational (on port 7001). Shortly will follow a blog entry to change the port for the Endeca Server.
6. Create a Weblogic Domain for Endeca Studio
For the Endeca Studio application we also need to create a domain (recommended as per the documentation). This is similar to the Endeca Server Domain creation: Open a command prompt and go to the directory:
<Weblogic Middleware Home>\wlserver_10.3\common\bin (for my environment:
d:\oracel\oeid30\fmw\wlserver_10.3\common\bin). Type there:
Select "Create a new WebLogic domain". Click <Next>. Use the default Basic domain configuration:
Entera Domain name and leave the Domain location untouched:
Click <Next>, Then enter a username and password for the domain administrator:
Click <Next>. Since we did a basic domain installation of the Endeca Server this has been configured on port 7001. So we need to change the port of the domain where Endeca Studio will be running. Change the Listen port of the future Endeca Studio Application. I chose port 8880.
Click <Next>. Then click <Create> and after a couple of seconds the domain has been created. Click <Done>.
A domain has been created and is ready to deploy the Studio application.
7. Update some Weblogic settings for the purpose of Endeca Studio
Before you can deploy the Endeca Studio application, we need to modify some Weblogic settings. Edit the file:
<Weblogic Middleware Home>\user_projects\domains\<Endeca Studio Domain>\bin\setDomainEnv.cmd
Copy the italics text below and paste it in the top of the file, i.e. just after set WL_HOME.
set JAVA_OPTIONS=-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
In the same file update (2 times) memory arguments:
Change set WLS_MEM_ARGS_64BIT=-Xms256m -Xmx512m into: set WLS_MEM_ARGS_64BIT=-Xms256m –Xmx1024m
set MEM_MAX_PERM_SIZE_64BIT=-XX:MaxPermSize=256m into: set MEM_MAX_PERM_SIZE_64BIT=-XX:MaxPermSize=512m
Save the file.
Stop the Endeca Studio domain if started (via command
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin\stopWebLogic.cmd) and start it (via command
<Weblogic Middleware Home>\user_projects\domains\<endeca server domain name>\bin\startWebLogic.cmd).
8. Deploy the Endeca Studio application
Now you will deploy the Endeca Studio application in the just created Weblogic domain. After you unzip the edelivery Studio package for Weblogic you'll see some files:
Copy the files
<Weblogic Middleware Home>\user_projects\domains
In the directory
<Weblogic Middleware Home>\user_projects\domains create the following directories: data, deploy and weblogic-deploy. In the newly created directory <Weblogic Middleware Home>\user_projects\domains\data create the directory endeca-data-sources.
The directory <Weblogic Middleware Home>\user_projects\domains will look like:
Now let's deploy the Endeca Studio file (endeca-portal-weblogic-3.0.10089.ear).
Make sure your Endeca Studio domain has been started. Open a browser and go to URL: http://localhost:<port number>/console (for me: http://localhost:8880/console).
Go to Configure applications:
Click Lock & Edit:
Locate the file endeca-portal-weblogic-3.0.10089.ear in
<Weblogic Middleware Home>\user_projects\domains and paste the path to that file in the Path space. Hit <enter>:
Again leave the defaults:
Click <Finish>. After the deployment is finished you will return to the overview screen. Click on <Activate Changes>:
Then the message appears that no restart is necessary:
Then click on deployments in the left pane of the Domain Structure. In the deployments then check endeca-portal-weblogic and from the Start drop down select: Servicing All Requests:
On the next screen click <Yes>. When the deployment screen returns and the State has been changed to Active, you can check the URL: http://localhost:<port number> (in my case: http://localhost:8880):
You can log on with username: firstname.lastname@example.org and password: Welcome123. You have to reset the password immediately after log on.
That was the install of Endeca Studio.
9. Install Integrator
The last piece of software before we can create our projects is the Integrator. After the unzip of the downloaded file you will see 3 files. 2 Files are needed when you would want to deploy the server version for production environments. For now double click the file EID_3.0_Integrator.exe.
Click <Next>, Click <Next> and click <Finish>.
That was the Integrator install.
Hopefully the next release of Endeca will give us the 'next, next, finish' experience again ...
Enjoy Endeca 3.0
Oracle Endeca Information Discovery 3.0 is out!. It is just released on 22nd March.
Some New Features:
- Refreshed Look And Feel
- New Chart Types (Bubble and Scatter)
- Lots of components upgraded (Metric Bar, Range Filter, Guided Navigation)
Endeca Server 7.5
- Language sensitive search for 22 languages
- New EQL features (like case statement, string join, etc.)
Data Integrator 3.0
- Some General ETL improvements (like job flow control)
- Enhanced support for some Data Sources (Spreadsheet, Microsoft Access)
- New Native JSON Reader
- New HTTPConnector
Something completely new and exciting is the Provisioning Services: The Provisioning Service enables dynamic application creation using data uploaded from the user's desktop. So that means that business users can upload their own data to the server and create their applications with no effort from the IT department.
Something we seem to have lost is the quick installation package. This means we have to do the 3 step installation of: Endeca Server, Integrator and Studio separately.
Also excited to get your hands on this new release? Go to https://edelivery.oracle.com
A new feature with OEID v2.4 is to load data directly from an Oracle Business Intelligence (OBI) Server. For companies who are using OBI and have modeled their common enterprise information model in it this can be a very nice feature. One place to get the structured data from!
How do we load data from OBI into an index with Integrator? According to the documentation we go in the Integrator to the menu bar and choose: File > New > Project. Integrator will display the New Project dialog. Then expand the Information Discovery Node and select Load Data from OBI Server.
Unfortunately this does not appear in my screen:
I can only see the Load Metadata from a Record Store ...
This perhaps has to do with my quickstart installation? It appears that not all plug-ins are present. To fix this download the Integrator v2.4 from edelivery.oracle.com. You will find it in the Media Pack: Oracle Endeca Information Discovery Integrator (2.4) Media Pack for Microsoft Windows x64 (64-bit) .
Unzip the file. You will see the plug-in appear:
Copy this file to the plug-ins directory of Integrator:
<Endeca Install Dir>\Endeca\Discovery\2.4.0\Integrator\plugins
Restart Integrator and the Wizard will appear as stated in the documentation:
You can now follow the instructions in the Wizard documentation to load data from the OBI server. I suggest the first time to choose the option: 'Create new project'. Then it becomes clear how it works (no magic there) and we can use it in other projects.
When you choose the option 'Create new project' a complete project will be generated. If you run the graph Baseline a new index will be created and data loaded from the OBI Server. The 2 important and re-usable information pieces are: How the format of the BI Server query should look (the QueryStatement.sql in the Navigator pane) and what connection information is needed (Connections in the Outline pane):
First have a look at the connection. If you double click the connection in the Outline pane the connection pane opens:
As you can see it uses the Oracle BI JDBC driver which is embedded in the Plug-in. So if we want to create a connection in another project we have to choose the driver
oracle.bi.jdbc.AnaJdbcDriver and copy (or type in) the URL
jdbc:oraclebi://wvillano-nl:9703/ (where wvillano-nl should be your OBI Server name) upon creation of the connection.
What should be the format of a query against the OBI Server? Double click on the QueryStatement.sql:
It looks like the logical query OBI is normally producing. But there is a difference. It starts with:
SELECT_BUSINESS_MODEL. And that is also what it does! So the entities should match the Business Model. Here a piece of my rpd:
As you can see in my presentation layer the
Department is still called
DEPT and in my Business Model layer it is called
Department. The Integrator query uses the Business Model naming:
So be aware, if your presentation layer has different names for attributes/tables then in your business model, you cannot copy/paste the logical query created in the OBI Server log. Otherwise: just copy/paste the logical query put the command SELECT_BUSINESS_MODEL before it and it will run.
We can also create any query we want manually of course e.g. add filters to the query:
In the generated project the query is stored in a seperate file, but you can also enter it as
'SQL Query' in the DBInputTable Reader component in the graph.
For the edge metadata we can use the same query since we are not able (yet) to create the metadata via the graphical interface.
That should be enough information to use OBI Server as a source in your projects. I hope you like it too.