ElasticSearch Copy index / reindex

First create the new index (with correct mapping, settings, etc) and then use reindex to copy the data. (See note at end for creating index with all settings like mappings etc)

POST _reindex
{
  "source": {
    "index": "old-index-name"
  },
  "dest": {
    "index": "new-index-name"
  }
}

In kibana, the above reindex command might return with error “Request failed to get to the server (status code: 504)”

Looks like it just timesout getting the response back from the server.

The stackoverflow link suggests looking at the status using

GET _tasks?detailed=true&actions=*reindex

This command just returned – since it had already completed

{
"nodes": {}
}

For a second reindex option when it timeout in Kibana, was able to see the status after the timeout error

  GET _tasks?detailed=true&actions=*reindex

  "nodes": {
    "LkcjzXLwT1-sv1UcmV9d6Q": {
      "name": "LkcjzXL",
      "roles": [
        "master",
        "data",
        "ingest"
      ],
      "tasks": {
        "Lkcj12345-sv1UcmV9d6Q:19511641": {
          "node": "Lkcj12345-sv1UcmV9d6Q",
          "id": 19511641,
          "type": "transport",
          "action": "indices:data/write/reindex",
          "status": {
            "total": 109850,
            "updated": 0,
            "created": 40000,
            "deleted": 0,
            "batches": 41,
            "version_conflicts": 0,
            "noops": 0,
            "retries": {
              "bulk": 0,
              "search": 0
            },
            "throttled_millis": 0,
            "requests_per_second": -1,
            "throttled_until_millis": 0
          },
          "description": "reindex from [old-index-name] to [new-index-name]",
          "start_time_in_millis": 1527792986092,
          "running_time_in_nanos": 116998993367,
          "cancellable": true,
          "headers": {}
        }
      }
    }
  }
}

 

Note: the reindex only copies data, to create an index with the same settings as an existing index, there seem to be two options – using templates or use snapshot/restore

See https://stackoverflow.com/questions/39274181/elasticsearch-how-to-make-a-1-to-1-copy-of-an-existing-index

 

Ref:

Advertisements

python – elasticsearch – root certificate error

elasticsearch – worked in some instances and not in others.

Failed in virtualenv and Pycharm – reason – missing certfi package. certfi was installed globally so when running in global context worked fine but not inside virtualenv.

installing certfi package fixed issue

(https://stackoverflow.com/a/43872382)

python Convert camelCase

Use python str.title() that converts string to TitleCase, then change the first character to lowercase;  if needed ignore characters that are not alphanumeric. Using this to convert header names in a csv file to be used as property names of an object

def to_camel_case(s):
    ret = ''.join(x for x in s.title() if x.isalpha())
    return ret[0].lower() + ret[1:]

Stackoverflow: see posts for other options and any latest info

DD-WRT, OpenVPN, Policy Based Routing, Excluding IPs/Netflix

These are just notes to use policy based routing when I decide to do so.

hopefully sometime in future setup rules to

  • send all traffic through vpn except for a couple of ips
  • send all traffic through vpn except for certain destinations like maybe netflix

related links to lookup


just random notes from looking at different posts:

  • openvpn adds rules new table ‘table 10’
  • ssh to the router and /tmp/openvpncl – has all the files
    • policy_ips has the ips we list in the ui
    • route_up.sh is the script that sets up the route table 10 and the routes for the ips
  • essentially routes will have to setup and/or firewall rules will have to be setup
  • routing
    • From the Linux networking help pages (http://linux-ip.net/html/routing-tables.html) found that there could be other routing tables other than main and the tables will be under /etc/iproute2/rt_tables. this is not present in the ddwrt. I couldn’t find a command that will list the different routing tables or where they are specified in dd-wrt.
    • “ip route” or “ip route show” lists all the routes from the main routing table
    • finally while looking at the help realized “ip route show table all” – will list the routes from all the tables, this is how i was able to see table 10 added by openvpn
    • ip rule list will show the different rules
    • you can also you “netstat -rn” to look at the rules (this will show the tun1 interface) and iptables to look at the firewall rules, “iptables -L” will list all the firewall rules

Netgear R6250 Advanced Tomato to Stock

After running into 2.4Ghz with Advanced Tomato decided to go back to the stock. Found the following links on the web.

Video of “Back to Netgear Genie” – http://tomato.groov.pl/?p=616

These are the steps I followed.

  1. Went to Tomato by Shibby page http://tomato.groov.pl/?page_id=164; here selected K26ARM;
  2. On the K26ARM page used the files under “Netgear R-Series back to OFW”
  3. Logged into my tomato, reset to default configuration with erasing NVRAM
  4. After that connected to one of the default tomato wifi networks, logged in again and upgraded the firmware using the downloaded file file R6250-back-to-ofw.trx
  5. After the upgrade completes, connected to one of the default netgear wifi networks (NETGEAR99 or NETGEAR99-5G), the wifi password is the one written on your router, sent from the factory. All set now back to configuring the network or should i switch to ddwrt again?

(I prefer doing all this over a wired connection instead of upgrading via wifi but too tired to care now.)

Netgear R6250 Advanced Tomato and 2.4Ghz issue

As of 2017-Jan there is an issue with 2.4Ghz band on the R6250 with the Advanced Tomato firmware. I also found this post https://www.myopenrouter.com/forum/what-hell-happened

I lost some static ip entries while using dd-wrt due to the known issue of how the page UI is very old and not really user friendly (great firmware though) – this drove me nuts so decided to try the Advanced Tomato on the R6250 – great UI. The Advanced Tomato UI worked great, firmware was great except for the 2.4Ghz band; the router page will show that the Wifi is good but none of the devices will be able to connect, tried various settings power etc, will show up then disappear and devices will not be able to connect. Searching the web brought up the page on myopenrouter listed above discussing the same issue. Switching back to stock/genie and maybe ddwrt.

Netgear R6250 reset after Advanced Tomato

Decided to try tomato on the Netgear R6250 since it has a nice UI. Upgraded from kong-dd-wrt to the latest stable download for Advanced Tomato.

There is an instruction on the tomato page that says, BEFORE UPGRADE you need to telnet to the router and issue a command to get the http password, I decided not to do that since I was going to reset after upgrade.  (http://www.polarcloud.com/tomatofaq#why_doesnt_my_password_work)

The factory/hard reset for Netgear R6250 as per the user manual is a simple 7 second press of the reset button on the back, of course i didnt remember this and did not bother to look up. I just pressed the reset button for more than 20seconds but the router did not reset, then I tried the 30-30-30 reset which did NOT work either. Instead a simple press of the reset button for about 7-10 seconds is all it needs!

(A little knowledge is a dangerous thing or in my case was a lot of wasted time 🙂   )

(Btw, the tomato firmware has 2.4Ghz issues)