elasticsearch-file-ingest

Streams and transforms large data files into Elasticsearch, handling NDJSON, CSV, Parquet, and Arrow IPC formats. The stream-based design processes large files without exhausting memory and reaches high throughput of 50k+ documents per second on commodity hardware, making it suited to loading files or batch-importing data rather than reindexing or general pipeline design. The skill is self-contained: its scripts folder and package.json live in the skill directory, dependencies are installed with npm install, and it requires Elasticsearch 8.x or 9.x plus Node.js 22+. Connection settings are supplied exclusively through environment variables, never as command-line arguments, covering Elastic Cloud (ELASTICSEARCH_CLOUD_ID plus ELASTICSEARCH_API_KEY), a direct URL with API key, basic authentication, and local development, with an optional ELASTICSEARCH_INSECURE flag to skip TLS verification in development. A connection test, node scripts/ingest.js test, should always run first, and ingestion should not proceed until it succeeds. Ingestion is driven by node scripts/ingest.js ingest with a required --target index and a source chosen from --file, which supports wildcards like logs/*.json for batch imports, or --stdin. Source format is set with --source-format (ndjson, csv, parquet, arrow). Mappings can be supplied via --mappings or inferred with --infer-mappings, which sends a sample to Elasticsearch's _text_structure/find_structure endpoint to derive both mappings and an ingest pipeline; importantly, --infer-mappings must not be combined with --source-format csv, or the index ends up empty. Additional options control CSV parsing, pipelines, index deletion, buffer size, progress display, and stall warnings. Custom JavaScript transform functions passed with --transform modify documents during ingestion: a function can enrich a document (for example adding a full_name field), skip it by returning null or undefined, or split one source document into many targets by returning an array. Custom index mappings are provided as JSON. Together these let files be cleaned, reshaped, and enriched on the way into Elasticsearch.