Solr Cloud Mode on FREEBSD

-

1.Set Bash as the default shell.

2.Create the groups and users


zookeeper:*:962:
solr:*:161:

...

zookeeper:*:962:962:zookeeper user:/nonexistent:/usr/sbin/nologin
solr:*:161:161:Apache Solr System:/var/db/solr:/usr/sbin/nologin

3.Download zookeeper and solr from https://zookeeper.apache.org/releases.html#download https://lucene.apache.org/solr/downloads.html and unzip them.


curl -O https://xxxxxx
OR

wget https://xxxxxx

4.link path to /usr/local/etc


tar zxf xxxx.gz
sudo ln -s /usr/local/etc/solr ~/solr.xx.xx
sudo ln -s /usr/local/etc/zookeeper ~/zookeeper.xx.xx

5.Customize solr config
copy solr/server/solr/configsets/_default to solr/server/solr/configsets/xxx_default

and
create file "data-config.xml"

<dataconfig>
  <datasource driver="org.postgresql.Driver" url="jdbc:postgresql://localhost/postgres" user="pgu1" password="pgu1" />
    <document>
      <entity name="item" query="select name as name_sxxx, cardno as cardno_sxxx, descriot as descriot_sxxx, ctftp as ctftp_sxxx, ctfid as ctfid_sxxx, gender as gender_sxxx, birthday as birthday_sxxx, address as address_sxxx, zip as zip_sxxx, dirty as dirty_sxxx, district1 as district1_sxxx, district2 as district2_sxxx, district3 as district3_sxxx, district4 as district4_sxxx, district5 as district5_sxxx, district6 as district6_sxxx, firstnm as firstnm_sxxx, lastnm as lastnm_sxxx, duty as duty_sxxx, mobile as mobile_sxxx, tel as tel_sxxx, fax as fax_sxxx, email as email_sxxx, nation as nation_sxxx, taste as taste_sxxx, education as education_sxxx, company as company_sxxx, ctel as ctel_sxxx, caddress as caddress_sxxx, czip as czip_sxxx, family as family_sxxx, version as version_sxxx, id from xxx" />
    </document>
</dataConfig>

.Update file "managed-schema" (optional?)


<schema name="example-basic" version="1.6">
=>
<schema name="xxx-data-driven-schema" version="1.6">

.Uncomment the flowlling line (optional?)


<!-- <copyfield source="*" dest="_text_"/> -->
=>
<copyfield source="*" dest="_text_"/>

.Update file "solrconfig.xml"

add the following lines to "solrconfig.xml"


  <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />




  <requesthandler name="/dataimport" class="solr.DataImportHandler">
    <lst name="defaults"><str name="config">data-config.xml</str></lst>
  </requestHandler>

and Uncommend the following lines to enable schemaless mode (optional?)



  <!-- This enabled schemaless mode 
  <initparams path="/update/**">
    <lst name="defaults">
      <str name="update.chain">add-unknown-fields-to-the-schema</str>
    </lst>
  </initParams>
  -->
=>
  <initparams path="/update/**">
    <lst name="defaults">
      <str name="update.chain">add-unknown-fields-to-the-schema</str>
    </lst>
  </initParams>

6.Zookeeper setting.

copy zoo_sample.cfg to zoo1.cfg, zoo2.cfg, zoo3.cfg.

update those cfg files

dataDir=/var/db/zookeeper/z1
clientPort=2181

4lw.commands.whitelist=mntr,conf,ruok
server.1=127.0.0.1:1888:18888
server.2=127.0.0.1:2888:28888
server.3=127.0.0.1:3888:38888


dataDir=/var/db/zookeeper/z2
clientPort=2182

4lw.commands.whitelist=mntr,conf,ruok
server.1=127.0.0.1:1888:18888
server.2=127.0.0.1:2888:28888
server.3=127.0.0.1:3888:38888


dataDir=/var/db/zookeeper/z3
clientPort=2183

4lw.commands.whitelist=mntr,conf,ruok
server.1=127.0.0.1:1888:18888
server.2=127.0.0.1:2888:28888
server.3=127.0.0.1:3888:38888

and create 3 myid files 


touch /var/db/zookeeper/z1/myid
touch /var/db/zookeeper/z2/myid
touch /var/db/zookeeper/z3/myid

echo "1" > /var/db/zookeeper/z1/myid
echo "2" > /var/db/zookeeper/z2/myid
echo "3" > /var/db/zookeeper/z3/myid


Create the following files
/usr/local/etc/rc.d
zookeeper1
zookeeper2
zookeeper3


#!/bin/sh

#
# PROVIDE: zookeeper
# REQUIRE: LOGIN
# KEYWORD: shutdown
#
# Add the following line to /etc/rc.conf to enable zookeeper:
#
# zookeeper1{2,3}_enable="YES"

. /etc/rc.subr

name=zookeeper1{2,3}
rcvar=zookeeper1{2,3}_enable

load_rc_config "${name}"

: ${zookeeper_enable:=NO}
: ${zookeeper_user:=zookeeper}
: ${zookeeper_group:=zookeeper}
: ${zookeeper_pidfile:=/var/run/zookeeper.pid}
: ${zookeeper_config:=/usr/local/etc/zookeeper/zoo1{2,3}.cfg}
: ${zookeeper_log4jpropfile:=file:/usr/local/etc/zookeeper/log4j.properties}
: ${zookeeper_rootlogger:="INFO,ROLLINGFILE"}
: ${zookeeper_logdir:=/var/log/zookeeper}
: ${zookeeper_jvmopts:=}

JAVA=/usr/local/openjdk8/bin/java
JAVAJARDIR=/usr/local/share/java/classes

for i in zookeeper jline slf4j-api netty log4j
do
    CLASSPATH="${JAVAJARDIR}/$i.jar:$CLASSPATH"
done
CLASSPATH=/usr/local/share/slf4j/slf4j-log4j12.jar:$CLASSPATH

zookeeper_main="-cp $CLASSPATH org.apache.zookeeper.server.quorum.QuorumPeerMain ${zookeeper_config}"
log4j_params="-Dzookeeper.log.dir=${zookeeper_logdir} \
            -Dlog4j.configuration=${zookeeper_log4jpropfile} \
            -Dzookeeper.root.logger=${zookeeper_rootlogger}"
pidfile=${zookeeper_pidfile}
procname=${JAVA}
required_files="${zookeeper_config}"

command="/usr/sbin/daemon"
command_args="-p ${pidfile} ${JAVA} ${zookeeper_jvmopts} ${log4j_params} ${zookeeper_main}"

start_precmd="zookeeper_init"

zookeeper_init()
{
    touch ${pidfile}
    chown ${zookeeper_user}:${zookeeper_group} ${pidfile}
    install -d -m 755 -o ${zookeeper_user} -g ${zookeeper_group} ${zookeeper_logdir}
}

run_rc_command "$1"
Add the following 3 lines to /etc/rc.conf to enable zookeeper:


zookeeper1_enable="YES"
zookeeper2_enable="YES"
zookeeper3_enable="YES"

start zookeeper service


sudo service zookeeper1 start
sudo service zookeeper2 start
sudo service zookeeper3 start

create a solr config path


[zk: localhost:2181(CONNECTED) 1] ls
ls [-s] [-w] [-R] path
[zk: localhost:2181(CONNECTED) 2] ls /
[aaabbbccc, zookeeper]
[zk: localhost:2181(CONNECTED) 3] create /solr
Created /solr
[zk: localhost:2181(CONNECTED) 4] create /solr/conifgs
Created /solr/conifgs
[zk: localhost:2181(CONNECTED) 5] ls
ls [-s] [-w] [-R] path
[zk: localhost:2181(CONNECTED) 6] ls /
[aaabbbccc, solr, zookeeper]
[zk: localhost:2181(CONNECTED) 7]

.Upload solr config files to zookeeper

zkcli.sh -cmd upconfig -n /solr/configsets/xxx -d solr/server/solr/configsets/xxx_default -z localhost:2181,localhost:2182,localhos:2183

7.Solr Setting
/usr/local/etc/rc.d
service files creating
solr1

#!/bin/sh


# PROVIDE: solr
# REQUIRE: LOGIN
# KEYWORD: shutdown

. /etc/rc.subr

name=solr1
rcvar=solr1_enable
load_rc_config $name

: ${solr1_enable:=NO}
: ${solr1_instance:=/data/solr/cloud/node1/solr}

solr_start ()
{
 su -m solr -c "${command} start"
}

solr_stop ()
{
 su -m solr -c "${command} stop"
}

solr_status ()
{
 su -m solr -c "${command} status"
}

# add /usr/local/bin to path
export PATH=$PATH:/usr/local/bin
# let the start script read some custom settings
export SOLR_INCLUDE=/usr/local/etc/solr1.in.sh

required_files="${solr1_instance}/solr.xml /usr/local/etc/solr1.in.sh ${solr1_instance}/log4j.properties"
command=/usr/local/solr/bin/solr
start_cmd=solr_start
stop_cmd=solr_stop
status_cmd=solr_status

run_rc_command "$1"

solr2

#!/bin/sh
#
# PROVIDE: solr
# REQUIRE: LOGIN
# KEYWORD: shutdown

. /etc/rc.subr

name=solr2
rcvar=solr2_enable
load_rc_config $name

: ${solr2_enable:=NO}
: ${solr2_instance:=/data/solr/cloud/node2/solr}

solr_start ()
{
 su -m solr -c "${command} start"
}

solr_stop ()
{
 su -m solr -c "${command} stop"
}

solr_status ()
{
 su -m solr -c "${command} status"
}

# add /usr/local/bin to path
export PATH=$PATH:/usr/local/bin
# let the start script read some custom settings
export SOLR_INCLUDE=/usr/local/etc/solr2.in.sh

required_files="${solr2_instance}/solr.xml /usr/local/etc/solr2.in.sh ${solr2_instance}/log4j.properties"
command=/usr/local/solr/bin/solr
start_cmd=solr_start
stop_cmd=solr_stop
status_cmd=solr_status

run_rc_command "$1"
solr1.in.sh


ZK_HOST="localhost:2181/solr,localhost:2182/solr,localhost:2183/solr"
SOLR_HOME="/data/solr/cloud/node1/solr"
LOG4J_PROPS="/data/solr/cloud/node1/solr/log4j.properties"
SOLR_LOGS_DIR="/data/solr/cloud/node1/logs"
SOLR_PORT="8983"
SOLR_PID_DIR="/var/db/solr"
#SOLR_OPTS="$SOLR_OPTS -Djetty.host=localhost"
SOLR_JAVA_MEM="-Xms1g -Xmx1g"
SOLR_SSL_KEY_STORE=/usr/local/etc/solr-ssl.keystore.jks
SOLR_SSL_KEY_STORE_PASSWORD=secret
SOLR_SSL_KEY_STORE_TYPE=JKS
SOLR_SSL_TRUST_STORE=/usr/local/etc/solr-ssl.keystore.jks
SOLR_SSL_TRUST_STORE_PASSWORD=secret
SOLR_SSL_TRUST_STORE_TYPE=JKS
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false

solr2.in.sh


ZK_HOST="localhost:2181/solr,localhost:2182/solr,localhost:2183/solr"
SOLR_HOME="/data/solr/cloud/node2/solr"
LOG4J_PROPS="/data/solr/cloud/node2/solr/log4j.properties"
SOLR_LOGS_DIR="/data/solr/cloud/node2/logs"
SOLR_PORT="8080"
SOLR_PID_DIR="/var/db/solr"
#SOLR_OPTS="$SOLR_OPTS -Djetty.host=localhost"
SOLR_JAVA_MEM="-Xms1g -Xmx1g"
SOLR_SSL_KEY_STORE=/usr/local/etc/solr-ssl.keystore.jks
SOLR_SSL_KEY_STORE_PASSWORD=secret
SOLR_SSL_KEY_STORE_TYPE=JKS
SOLR_SSL_TRUST_STORE=/usr/local/etc/solr-ssl.keystore.jks
SOLR_SSL_TRUST_STORE_PASSWORD=secret
SOLR_SSL_TRUST_STORE_TYPE=JKS
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false

Add the following lines to /etc/rc.conf to enable Solr:


solr1_enable="YES"
solr2_enable="YES"

Create 2 Solr node folders
/data/solr/cloud
node1/solr
node2/solr
.Copy solr-x.x.x/server/solr/solr.xml to
/data/solr/cloud/node1/solr
and
/data/solr/cloud/node2/solr

create log4j.properties file to

/data/solr/cloud/node1/solr
and
/data/solr/cloud/node2/solr


# Default Solr log4j config
# rootLogger log level may be programmatically overridden by -Dsolr.log.level
solr.log=/data/solr/cloud/node1{2}/logs
log4j.rootLogger=WARN, file, CONSOLE

# Console appender will be programmatically disabled when Solr is started with option -Dsolr.log.muteconsole
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n

#- size rotation with log cleanup.
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.MaxFileSize=4MB
log4j.appender.file.MaxBackupIndex=9

#- File to log to and log format
log4j.appender.file.File=${solr.log}/solr.log
log4j.appender.file.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n

# Adjust logging levels that should differ from root logger
log4j.logger.org.apache.zookeeper=WARN
log4j.logger.org.apache.hadoop=WARN
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.server.Server=INFO
log4j.logger.org.eclipse.jetty.server.ServerConnector=INFO

# set to INFO to enable infostream log messages
log4j.logger.org.apache.solr.update.LoggingInfoStream=OFF


At the last start the solr services.


sudo service solr1 start
sudo service solr2 start

8.Creatting Collection


solr create_collection -c xxx -n xxx_config -shards 2 -replicationFactor 2 -p 8983

9.Importting data ..etc

Omitted.

Comments

Post a Comment

Popular posts from this blog

Python3 + Django + uWSGI + Nginx On FreeBSD

-
1. uWSGI Installation. $ sudo pip-3.6 install uwsgi Collecting uwsgi Downloading uwsgi-2.0.17.tar.gz (798kB) 100% |################################| 798kB 709kB/s Installing collected packages: uwsgi Running setup.py install for uwsgi ... done Successfully installed uwsgi-2.0.17 1.1 Basic test Let’s start with a simple “Hello World” example: def application(env, start_response): start_response('200 OK', [('Content-Type','text/html')]) return [b"Hello World"] As you can see, it is composed of a single Python function. It is called “application” as this is the default function that the uWSGI Python loader will search for (but you can obviously customize it). 1.2 Now start uWSGI to run an HTTP server/router passing requests to your WSGI application: uwsgi --http :9090 --wsgi-file foobar.py 2.Installing Django. Install Django into your virtualenv , create a new project $ mkproject django $ pip install Django Collecting Django ...
Read more

arduino最小構成 (atmega328/8MHz/3.3V/内部クロック)FT232RLにてブートローダーの書き込み

-
Image
１．部品 ATMEGA328P-PU USBシリアル変換モジュールFT232RL 0.1μFコンデンサ ブレッドボード ジャンパーワイヤ ２．ROM,DRIVER,書き込み用ソフトの準備 breadboard-1-6-x.zip をダウンロードして展開します。 FTAVRW : FT232-AVR Writerの ftavrw_v121b.zip をダウンロードして展開します。 https://www.ftdichip.com/Drivers/VCP.htm FTDIドライバをダウンロードして展開、インストールします。 ３．ブートローダーの書き込み 配線 下図はブレッドボード上に配線した例です。 ftavrwg.exeをクリックして実行します。 avr info ボタンを押下し、情報が取得されたら配線はOKです。 ATmegaBOOT_168_atmega328_pro_8MHz.hexを選択し、書き込みます。 下図のようなメッセージが出ましたら、書き込みが完了です。 boards.txtを確認して、FUSEを書き込みます。 下図のようなメッセージが出ましたら、書き込みが完了です。 ４．スケッチの書き込み 配線 下図はスケッチの配線例です。注意： RESET に 0.1μF のコンデンサを噛ます必要があります arduino IDEを開いてexamples(Blink)を書き込んでみます。 書き込みは無事にできました。Blinkもちゃんと動いています。
Read more

FreeBSD: Configuring Apache to permit CGI

-
In httpd.conf file make sure the "cgi" LoadModule directive has not been commented out. A correctly configured directive may look like this: <IfModule mpm_prefork_module> LoadModule cgi_module libexec/apache24/mod_cgi.so </IfModule> Explicitly using Options to permit CGI execution To allow CGI program execution for any file ending in .cgi in users' directories, you can use the following configuration. <VirtualHost *:80> DocumentRoot "/usr/local/www/apache24/e" ServerName www.example.com ErrorLog "/var/log/e-error_log" CustomLog "/var/log/e-access_log" common <Directory "/usr/local/www/apache24/e/"> Order allow,deny Allow from all Require all granted Options +ExecCGI AddHandler cgi-script .cgi </Directory> </VirtualHost> <Directory "/home/*/public_html"> Options +ExecCGI AddHandler cgi-script .cgi </Di...
Read more