This worklog has been replaced with mariadb.org/jira

This site is here for historical purposes only. Do not add or edit tasks here!

 
 
 

WorkLog Frontpage Log in / Register
High-Level Description | Task Dependencies | High-Level Specification | Low-Level Design | File Attachments | User Comments | Time Estimates | Funding and Votes | Progress Reports

 Add Sphinx storage engine to MariaDB
Title
Task ID42
Queue
Version
Status
Priority
Copies toSergei

Created byMonty10 Aug 2009Done
Supervisor   
Lead Architect    
Architecture Review 23 Jul 2010
Implementor 23 Jul 2010
Code Review 26 Jul 2010
QA  
Documentation 26 Jul 2010
 High-Level Description
Add the Sphinx storage engine to the MariaDB tree
 Task Dependencies
Others waiting for Task 42Task 42 is waiting forGraph
127 generalize mtr to support per-suite extensions
 
 High-Level Specification
Code
----

Andrew Aksyonoff from Sphinx is helping to integrate the SphinxSE plugin into
the MariaDB tree.

It is a plugin, so it can be added to the tree just by including the
sub-directory storage/sphinx/.

The Sphinx plugin is already of some maturity, having been used with MySQL for
some time.


Testing
-------

To get testing in the mysql-test-run framework, some extensions are needed.

To use the Sphinx storage engine, the external Sphinx search daemon needs to
be running with some data directory containing indexed data. It also needs to
be allocated a port.

This is the indended approach:

1. Testing will use an external Sphinx setup installed on the machine. Sphinx
binaries will be searched in typical locations (eg. /usr/bin, /usr/local/bin),
or can be specified explicitly in the environment with SPHINXSEARCH_INDEXER
and SPHINXSEARCH_SEARCHD for the two required binaries. If the external Sphinx
binaries can not be found, then Sphinx tests will be disabled (using some
--source include/have_sphinx.inc in the test cases).

2. The mysql-test-run framework will install Sphinx search data and start/stop
the Sphinx search daemon for the test cases, similarly how it is done for the
other servers mysqld, ndbd, etc. We will run the Sphinx search daemon with
options --console, --config, and --pidfile.

3. The mysql-test-run framework will generate a Sphinx config file from a
template in mysql-test/suite/sphinx/my.cnf. This config file will allocate
ports and data directories appropriate for avoiding conflicts between multiple
simultaneous mysql-test-run executions. The Sphinx config file is sufficiently
similar to MySQL my.cnf that we can use the existing framework for generating
config file, with just a slightly modified variant of the code writing the
file to disk.

4. The mysql-test-run framework will pre-load the mysql database with tables
and data for Sphinx to index. It will then run the `indexer` program to
generate the indexes, and then start the `searchd` daemon. These three steps
must be done in order, as each step depends on the previous. ALTERNATIVE: it
might be possible to pre-generate the necessary data/index files and store
them in the source tree.

I pushed a proof-of-concept patch for this here:

    lp:~knielsen/maria/5.2-sphinxse

Here is a sample test case using this:

--source include/have_sphinx.inc
--source include/have_sphinxse.inc

--replace_result $SPHINXSEARCH_PORT SPHINXSEARCH_PORT
eval create table ts ( id int unsigned not null, w int not null, q varchar(255)
not null, index(q) ) engine=sphinx
connection="sphinx://127.0.0.1:$SPHINXSEARCH_PORT/*";
select * from ts where q='test';
drop table ts;
 Low-Level Design
 File Attachments
 NameTypeSizeByDate
 User Comments
 Time Estimates
NameHours WorkedLast Updated
Knielsen907 Jun 2010
Sergei626 Jul 2010
Total15 
 Hrs WorkedProgressCurrentOriginal
This Task15016
Total15016
 
 Funding and Votes
Votes: 1: 0%
 Change vote: Useless    Nice to have    Important    Very important    

Funding: 0 offers, total 0 Euro
 Progress Reports
(Sergei - Mon, 26 Jul 2010, 09:40
    
code review, sphinxse fixes, looking at mtr changes - moved them to wl#127
Worked 6 hours and estimate 0 hours remain (original estimate decreased by 1 hour).

(Sergei - Mon, 26 Jul 2010, 08:48
    
Code Review signoff

(Sergei - Mon, 26 Jul 2010, 08:48
    
Status updated.
--- /tmp/wklog.42.old.14860	2010-07-26 08:48:40.000000000 +0000
+++ /tmp/wklog.42.new.14860	2010-07-26 08:48:40.000000000 +0000
@@ -1 +1 @@
-In-Progress
+Complete

(Sergei - Mon, 26 Jul 2010, 08:48
    
Documentation signoff

(Sergei - Mon, 26 Jul 2010, 08:45
    
Dependency created: WL#127 now depends on WL#42

(Sergei - Fri, 23 Jul 2010, 12:05
    
Status updated.
--- /tmp/wklog.42.old.2562	2010-07-23 12:05:29.000000000 +0000
+++ /tmp/wklog.42.new.2562	2010-07-23 12:05:29.000000000 +0000
@@ -1 +1 @@
-Assigned
+In-Progress

(Sergei - Fri, 23 Jul 2010, 12:05
    
Architecture Review updated:  -> Sergei
Architecture Review signoff
Implementor signoff
Code Review updated:  -> Sergei
Documentation updated:  -> Dbart

(Sergei - Thu, 01 Jul 2010, 06:20
    
Observers changed: Sergei

(Knielsen - Mon, 07 Jun 2010, 07:12
    
Help Andrew with the integration.
Worked 2 hours and estimate 7 hours remain (original estimate unchanged).

(Knielsen - Mon, 31 May 2010, 06:49
    
Wrote patch that allows to test SphinxSE in mysql-test-run, using external Sphinx daemon.
Worked 7 hours and estimate 9 hours remain (original estimate unchanged).
-- View All Progress Notes (18 total) --


Report Generator:
 
Saved Reports:

WorkLog v4.0.0
  © 2010  Sergei Golubchik and Monty Program AB
  © 2004  Andrew Sweger <yDNA@perlocity.org> and Addnorya
  © 2003  Matt Wagner <matt@mysql.com> and MySQL AB