Run NiFi Secure
I created an install script (linux machines) that will install a local secure version of NiFi. It will also generate your own certs for testing.
Please check the NiFi mirror site for latest release. Be sure to set the settings.txt file (ie. NIFI_VERSION=1.12.0)
The scripts will :
1. Create 3 self signed certs (admin, guest, test)
2. Install the latest version of NiFi (1.11.4)
3. Modify the nifi.properties file to run secure
4. Copy over a basic flow with proper permission for each cert
'Out of the box', everything is installed in a new directory (/tmp/nifi)
Steps:
1. Download and untar installNiFi.tar
2. run the bash script installNiFi.sh
3. /tmp/nifi-current/bin/nifi.sh start
4. tail the logs; tail -f /tmp/nifi-current/logs/*log
View your running nifi instance:
1. In a browser go to http://127.0.0.1:8080
2. Download the 3 certs (admin, guest, test)
3. In a browser go to https://127.0.0.1:8443
Now you have access to a secure installed version of nifi!
If you want to access the secure NiFi outside your local machine, edit the
/tmp/nifi-current/conf/nifi.properties and set the nifi.web.https.host value to your public IP address (ie. nifi.web.https.host=192.168.1.32)
Coming soon Install NiFi Cluster Secure
Silver Cloud Computing
.... The ONE silver lining for all of your software needs!
Expression Language
matches mp3 files ${filename:matches('.\.mp3.*)'}
AND match first/last name attributes ${firstName:equals('rob'):and(${lastname:equals('smith')})}
OR match first/last name attributes ${firstName:matches('rob|robert|bob')}
match 2 attributes ${attr1:equals(${attr2})}
${now():format("EEE"):matches('Sat|Sun')}
${filename:matches('[A-Z0-9]{12}.txt}
${filename:matches('..mike.*)}
grab whole line (^(.*?)$)
grab first 2 lines ^.*[\r\n].*
\n(.+)\n
extract first 3 characters ^.{0,3}
extract first 10 characters (^.{10}(.))
${anyMatchingAttribute(".*"):contains('mike')}
OR match xxxxxxxxxx name attributes ${filename:matches('me.*|mine.*'):and( ${filename:matches('.*MID.LE.*')} )}
check large (1MB) filesize ${fileSize:ge(1000000)}
remove invalid - from filename ${filename:replaceAll("[^a-zA-Z0-9\\.\\-\\-\\_","_")}
Delete Attribute (val1|val2|val3)
Keep only attributes ^((?!val1).)*
Does not start with ^(?!val1).*
!equals ^(?val1$).*
approved list (val1|val2|val3)
random values 1->10 ${random():mod(10):plus(1)}
set formatted date now():format('yyyy_MM_dd_HH_mm_ss')
id mime type ${mime.type:equals('application/bzip2')}
JSON
get last arrry item $.names[(@.legnth-1)
let first title $.store.book[0].title
get all titles $.store.book[*].title
get all prices $.store..price
get book 3 values $..book[3]
get price <10 $.store.book[?(@.price<10].title
get fiction books $.store.book[?(@.catagory=='fiction'].titleType your paragraph here.
"store": {
"book": [
{
"category": "reference",
"authors": [
{
"firstName": "Nigel",
"lastName": "Rees"
},
{
"firstName": "Evelyn",
"lastName": "Waugh"
}
],
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"authors": [
{
"firstName": "A",
"lastName": "B"
},
{
"firstName": "C",
"lastName": "D"
}
],
"title": "Sword of Honour",
"price": 12.99
},
{
"category": "fiction",
"authors": [
{
"firstName": "A",
"lastName": "D"
},
{
"firstName": "Evelyn",
"lastName": "X"
}
],
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"price": 8.99
},
{
"category": "fiction",
"authors": [
{
"firstName": "Nigel",
"lastName": "Rees"
},
{
"firstName": "Evelyn",
"lastName": "X"
}
],
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"price": 22.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
},
"expensive": 10
}
Parse Xml and JSON NiFi DataFlow
I created a NiFi template that you can use to help better understand how to parse xml and json data using several of the built in NiFi processors. The flow will query the USGS web site for the last 24 hours of earthquake data. The received (quake) XML file will be transformed into JSON data using the test.xslt file. Both the XML and JSON results will be parsed for pertinent quake info.
Steps :
1. Download and install the latest NiFi version
2. Download the ParseXmlJson (.xml) template
3. Download the test.xslt transform file and store it in your base NiFi directory
(ie. ...\nifi-1.11.4-bin\nifi-1.11.4\test.xslt)
4. Start NiFi
5. Import the template
6. Add the template to your NiFi canvas
7. Start the InvokeHttp processor to get some data.
NiFi Processors used:
EvaluateXPath
EvaluateXQuery
JoltTransformJSON
SplitJson
TransformXml