Embedded Files
Sources: https://www.elastic.co/, https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config.html
---------------------------INSTALLATION------------------------------------------------
Requires java 8/+
Download Each component(elasticsearch, kibana, logstash, beats, x-pack, elastic cloud) in Elastic Stack.
1> untar elasticsearch. >> set ES_HOME=extracted folder. >> append ES_HOME/bin to PATH. >> elasticsearch-plugin install x-pack. >> run elasticsearch
elasticsearch2> untar kibana. >> set KIBANA_HOME=extracted folder. >> append KIBANA_HOME/bin to PATH. >> kibana-plugin install x-pack. >> run kibana >> connect kibana at http://<server>:5601 >> default credentials are below.
kibana3>Elasticsearch 1.7.1, same as other version(Download> Extract> Set home> run.).
To setup marvel & sense:
/opt/es/elasticsearch-1.7.1/bin/plugin -i elasticsearch/marvel/latestBrowse: http://<HOST>:9200/_plugin/marvel/sense/index.html
---------------------------SECURITY------------------------------------------------
X-Pack security provides built-in user credentials to help you get up and running. These users have a fixed set of privileges and the default password changeme.
Table 3. X-Pack security Built-in Users
Built-in User Names Description
elastic A built-in superuser.
kibana The user Kibana uses to connect and communicate with Elasticsearch.
logstash_system The user Logstash uses when storing monitoring information in Elasticsearch.
----------------------------BASICS------------------------------------------------
A node is a running instance of Elasticsearch, while a cluster consists of one or more nodes with the same cluster.name that are working together to share their data and workload. As nodes are added to or removed from the cluster, the cluster reorganizes itself to spread the data evenly.One node in the cluster is elected to be the master node, which is in charge of manag‐ing cluster-wide changes like creating or deleting an index, or adding or removing anode from the cluster. The master node does not need to be involved in document-level changes or searches, which means that having just one master node will notbecome a bottleneck as traffic grows. Any node can become the master. As users, we can talk to any node in the cluster.
----------------------------Query------------------------------------------------
http://<ES_SERVER>:9200/<INDEX_NAME>/_search?q=<KEY>:<VALUE> ##e.g is belowhttp://localhost:9200/studs/_search?q=name:test----------------------------PROBLEMS & SOLUTIONS------------------------------------------------
Problem#1: Trial license expired. hence x-pack stopped working and kibana could not start without x-pack on ES.
Sol#: elasticsearch-plugin remove x-pack >> rm -rf $ES_HOME/config/x-pack >> elasticsearch-plugin install x-pack. Done.
Problem#2: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
Sol#: Node names were not defined in configuration.
Problem#3: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [detected node details]
Sol#: this was solved by 2 ways. 1> Removed x-pack as it was holding user credentials and was blocking client(without creds).
2> Appended .put("xpack.security.user", "<USER>:<PASSWORD>") to settings builder.
Problem#4:cluster service closed while waiting for enough shards to be started.
Sol#: Solved after restarting service. Seems process died due to lack of resource.
Problem#5: Java client API : class PreBuiltTransportClient was not found.
Sol#: Added dependency
compile group: 'org.elasticsearch.client', name: 'transport', version: '5.2.0'
Problem#6: Java client API : class PreBuiltXPackTransportClient (For X-pack secured ES) was not found.
Sol#: Added dependency
compile group: 'org.elasticsearch.client', name: 'x-pack-transport', version: '5.2.0'
and repository
maven{ url "https://artifacts.elastic.co/maven"}
Problem#7: ClassNotFoundException for org.apache.logging.log4j.Logger
Sol#: Added dependency
compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.7'
Problem#8: eclipse complained compile time failure due to version mismatch.
Sol#: Client API should be of same major version. [e.g- for ES 5.2, client should be atleast 5.0 + ]
Additionally JDK1.8 + was required to run.
Problem#9: org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
Sol#: switched to other non root user and then executed "elasticsearch". [hadoop in pig node]
Problem#10: Exception in thread "main" java.lang.UnsupportedClassVersionError: org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor version 52.0
Sol#: switched to java8 .
Problem#11: java.io.FileNotFoundException: /opt/es/elasticsearch-5.2.0/logs/es-cluster-1.log (Permission denied)
Sol#: log file was created by root user while fail attempts were made. manually deleted the file > switched user> then started es.
Problem#12:max number of threads [1024] for user [hadoop] is too low, increase to at least [2048]
Sol#:
ulimit -n 65536 #sets max number of open files to 65536.ulimit -u 2048 #sets number of threads.sysctl -w vm.max_map_count=262144 #for virtual memory limits. to be appended in /etc/sysctl.confProblem#13:storm bolt error while connecting . [elasticsearch[Prime][generic][T#3]] INFO o.e.c.transport - [Prime] failed to get node info for
Sol#: downgraded es instance to 1.7.1 -- compatible for java 7 & es-java-client used 1.7.1 .
Create and upload examples: https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html
https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.1/java-docs-index.html
http://teknosrc.com/execute-raw-elasticsearch-query-using-transport-client-java-api/
Examples: TransportClient Without X-Pack Installed, TransportClient With X-Pack Installed, NodeClient
Problem#14: Elasticsearch 7.6 : Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting.
Sol#: start cluster with ./elasticsearch-7.6.2/bin/elasticsearch -Esearch.max_buckets=1000000
Problem#15: Elasticsearch 6.0: *Bucket term Aggregation.. error while aggregating on text field.
#15.1 Fielddata is disabled on text fields by default. Set fielddata=true on [db] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead
#15.2 this occurrs when 4th level of term aggregation was applied
[request] Data too large, data for [<agg [by_event]>] would be [639019640/609.4mb], which is larger than the limit of [639015321/609.4mb]
-----------
Google Sites
Report abuse